2024-03-29T11:06:04.085 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T11:06:04.085 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622681 --name amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi --verbose --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} --owner scheduled_amathuri@teuthology -- /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622681/orig.config.yaml 2024-03-29T11:06:04.119 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:06:04.206 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622681 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} email: null first_in_suite: false job_id: '7622681' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8152 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi104.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMapDgIsdLYjW2H7cUSLyAA2wd86CeL/sKuJHvyFFC9a+OL//CvJkSpgxOVDA9BOb4KTT7tX67mxUCkvnffYm4g= smithi119.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM6RWsxLIJXsXGfKg2EOoww8oIOk8ov7+dbvExTnnCoD8eyyxNhyxmQFEDeoq7WrTYD4l6S+hXD66rx7Qe4rA34= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_05:45:44 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T11:06:04.206 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-29T11:06:04.207 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-29T11:06:04.207 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T11:06:04.209 INFO:teuthology.task.internal:Checking packages... 2024-03-29T11:06:04.233 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-29T11:06:04.233 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T11:06:04.233 INFO:teuthology.packaging:ref: None 2024-03-29T11:06:04.233 INFO:teuthology.packaging:tag: None 2024-03-29T11:06:04.234 INFO:teuthology.packaging:branch: main 2024-03-29T11:06:04.234 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:06:04.234 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-29T11:06:04.779 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2530.gb74786fc 2024-03-29T11:06:04.780 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T11:06:04.786 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T11:06:04.786 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T11:06:04.807 INFO:teuthology.task.internal:Saving configuration 2024-03-29T11:06:04.820 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T11:06:04.825 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:06:04.848 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi104.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622681', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:01:12.759016', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMapDgIsdLYjW2H7cUSLyAA2wd86CeL/sKuJHvyFFC9a+OL//CvJkSpgxOVDA9BOb4KTT7tX67mxUCkvnffYm4g='} 2024-03-29T11:06:04.871 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi119.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622681', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:01:12.757536', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM6RWsxLIJXsXGfKg2EOoww8oIOk8ov7+dbvExTnnCoD8eyyxNhyxmQFEDeoq7WrTYD4l6S+hXD66rx7Qe4rA34='} 2024-03-29T11:06:04.871 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T11:06:04.876 INFO:teuthology.task.internal:roles: ubuntu@smithi104.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-29T11:06:04.876 INFO:teuthology.task.internal:roles: ubuntu@smithi119.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-29T11:06:04.876 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T11:06:04.948 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff5022890d0>, signals=[15]) 2024-03-29T11:06:04.948 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T11:06:04.954 INFO:teuthology.task.internal:Opening connections... 2024-03-29T11:06:04.954 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi104.front.sepia.ceph.com 2024-03-29T11:06:04.956 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:06:05.089 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi119.front.sepia.ceph.com 2024-03-29T11:06:05.090 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:06:05.178 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T11:06:05.184 DEBUG:teuthology.orchestra.run.smithi104:> uname -m 2024-03-29T11:06:05.201 INFO:teuthology.orchestra.run.smithi104.stdout:x86_64 2024-03-29T11:06:05.201 DEBUG:teuthology.orchestra.run.smithi104:> cat /etc/os-release 2024-03-29T11:06:05.257 INFO:teuthology.orchestra.run.smithi104.stdout:NAME="CentOS Stream" 2024-03-29T11:06:05.257 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION="9" 2024-03-29T11:06:05.257 INFO:teuthology.orchestra.run.smithi104.stdout:ID="centos" 2024-03-29T11:06:05.257 INFO:teuthology.orchestra.run.smithi104.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:06:05.257 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION_ID="9" 2024-03-29T11:06:05.257 INFO:teuthology.orchestra.run.smithi104.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:06:05.257 INFO:teuthology.orchestra.run.smithi104.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:06:05.257 INFO:teuthology.orchestra.run.smithi104.stdout:ANSI_COLOR="0;31" 2024-03-29T11:06:05.257 INFO:teuthology.orchestra.run.smithi104.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:06:05.257 INFO:teuthology.orchestra.run.smithi104.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:06:05.258 INFO:teuthology.orchestra.run.smithi104.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:06:05.258 INFO:teuthology.orchestra.run.smithi104.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:06:05.258 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:06:05.258 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:06:05.258 INFO:teuthology.lock.ops:Updating smithi104.front.sepia.ceph.com on lock server 2024-03-29T11:06:05.284 DEBUG:teuthology.orchestra.run.smithi119:> uname -m 2024-03-29T11:06:05.302 INFO:teuthology.orchestra.run.smithi119.stdout:x86_64 2024-03-29T11:06:05.302 DEBUG:teuthology.orchestra.run.smithi119:> cat /etc/os-release 2024-03-29T11:06:05.359 INFO:teuthology.orchestra.run.smithi119.stdout:NAME="CentOS Stream" 2024-03-29T11:06:05.359 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION="9" 2024-03-29T11:06:05.359 INFO:teuthology.orchestra.run.smithi119.stdout:ID="centos" 2024-03-29T11:06:05.359 INFO:teuthology.orchestra.run.smithi119.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:06:05.359 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION_ID="9" 2024-03-29T11:06:05.359 INFO:teuthology.orchestra.run.smithi119.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:06:05.360 INFO:teuthology.orchestra.run.smithi119.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:06:05.360 INFO:teuthology.orchestra.run.smithi119.stdout:ANSI_COLOR="0;31" 2024-03-29T11:06:05.360 INFO:teuthology.orchestra.run.smithi119.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:06:05.360 INFO:teuthology.orchestra.run.smithi119.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:06:05.360 INFO:teuthology.orchestra.run.smithi119.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:06:05.360 INFO:teuthology.orchestra.run.smithi119.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:06:05.360 INFO:teuthology.orchestra.run.smithi119.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:06:05.360 INFO:teuthology.orchestra.run.smithi119.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:06:05.360 INFO:teuthology.lock.ops:Updating smithi119.front.sepia.ceph.com on lock server 2024-03-29T11:06:05.383 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T11:06:05.392 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T11:06:05.397 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T11:06:05.397 DEBUG:teuthology.orchestra.run.smithi104:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:06:05.400 DEBUG:teuthology.orchestra.run.smithi119:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:06:05.416 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T11:06:05.421 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T11:06:05.421 DEBUG:teuthology.orchestra.run.smithi104:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:06:05.454 DEBUG:teuthology.orchestra.run.smithi119:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:06:05.675 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T11:06:05.731 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T11:06:05.745 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T11:06:05.746 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T11:06:05.746 DEBUG:teuthology.orchestra.run.smithi104:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:06:05.746 DEBUG:teuthology.orchestra.run.smithi119:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:06:05.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:06:05.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:06:05.764 DEBUG:teuthology.orchestra.run.smithi119:> uname -r 2024-03-29T11:06:05.764 DEBUG:teuthology.orchestra.run.smithi104:> uname -r 2024-03-29T11:06:05.819 INFO:teuthology.orchestra.run.smithi104.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:06:05.819 INFO:teuthology.task.kernel:Running kernel on smithi104: 5.14.0-361.el9.x86_64 2024-03-29T11:06:05.819 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum install -y kernel 2024-03-29T11:06:05.821 INFO:teuthology.orchestra.run.smithi119.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:06:05.821 INFO:teuthology.task.kernel:Running kernel on smithi119: 5.14.0-361.el9.x86_64 2024-03-29T11:06:05.821 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum install -y kernel 2024-03-29T11:06:07.876 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-29T11:06:08.130 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-29T11:06:09.259 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS-9 - BaseOS 6.9 MB/s | 2.0 MB 00:00 2024-03-29T11:06:09.789 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-03-29T11:06:10.621 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-03-29T11:06:11.316 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - BaseOS 4.6 MB/s | 8.0 MB 00:01 2024-03-29T11:06:12.732 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - AppStream 18 MB/s | 19 MB 00:01 2024-03-29T11:06:13.226 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-03-29T11:06:17.346 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - CRB 9.1 MB/s | 6.1 MB 00:00 2024-03-29T11:06:17.688 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - CRB 4.1 MB/s | 6.1 MB 00:01 2024-03-29T11:06:18.786 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - Extras packages 52 kB/s | 16 kB 00:00 2024-03-29T11:06:19.147 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - Extras packages 49 kB/s | 16 kB 00:00 2024-03-29T11:06:20.042 INFO:teuthology.orchestra.run.smithi104.stdout:Extra Packages for Enterprise Linux 27 MB/s | 21 MB 00:00 2024-03-29T11:06:22.787 INFO:teuthology.orchestra.run.smithi119.stdout:Extra Packages for Enterprise Linux 5.3 MB/s | 21 MB 00:03 2024-03-29T11:06:24.873 INFO:teuthology.orchestra.run.smithi104.stdout:lab-extras 45 kB/s | 1.7 kB 00:00 2024-03-29T11:06:27.071 INFO:teuthology.orchestra.run.smithi104.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:06:27.071 INFO:teuthology.orchestra.run.smithi104.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:06:27.166 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-03-29T11:06:27.182 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-03-29T11:06:27.182 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-03-29T11:06:27.182 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-03-29T11:06:27.182 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-03-29T11:06:27.182 INFO:teuthology.orchestra.run.smithi104.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:06:27.182 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-03-29T11:06:27.182 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:06:27.183 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:06:27.183 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:06:27.183 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:06:27.183 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-03-29T11:06:27.183 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-03-29T11:06:27.183 INFO:teuthology.orchestra.run.smithi104.stdout:Install 4 Packages 2024-03-29T11:06:27.183 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:06:27.183 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 76 M 2024-03-29T11:06:27.184 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 123 M 2024-03-29T11:06:27.184 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-03-29T11:06:27.332 INFO:teuthology.orchestra.run.smithi104.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-03-29T11:06:27.795 INFO:teuthology.orchestra.run.smithi119.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-03-29T11:06:27.899 INFO:teuthology.orchestra.run.smithi104.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 21 MB/s | 15 MB 00:00 2024-03-29T11:06:28.608 INFO:teuthology.orchestra.run.smithi104.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 24 MB/s | 34 MB 00:01 2024-03-29T11:06:28.866 INFO:teuthology.orchestra.run.smithi104.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 18 MB/s | 28 MB 00:01 2024-03-29T11:06:28.870 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:06:28.870 INFO:teuthology.orchestra.run.smithi104.stdout:Total 45 MB/s | 76 MB 00:01 2024-03-29T11:06:28.870 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-03-29T11:06:29.242 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-03-29T11:06:29.242 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-03-29T11:06:30.035 INFO:teuthology.orchestra.run.smithi119.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:06:30.035 INFO:teuthology.orchestra.run.smithi119.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:06:30.089 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-03-29T11:06:30.089 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-03-29T11:06:30.119 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-29T11:06:30.128 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-29T11:06:30.129 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-03-29T11:06:30.129 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-29T11:06:30.129 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-03-29T11:06:30.129 INFO:teuthology.orchestra.run.smithi119.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:06:30.129 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-03-29T11:06:30.129 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:06:30.129 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:06:30.129 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:06:30.129 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:06:30.129 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-03-29T11:06:30.129 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-29T11:06:30.129 INFO:teuthology.orchestra.run.smithi119.stdout:Install 4 Packages 2024-03-29T11:06:30.129 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:06:30.130 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 76 M 2024-03-29T11:06:30.130 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 123 M 2024-03-29T11:06:30.130 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-03-29T11:06:30.271 INFO:teuthology.orchestra.run.smithi119.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-03-29T11:06:30.830 INFO:teuthology.orchestra.run.smithi119.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 22 MB/s | 15 MB 00:00 2024-03-29T11:06:31.330 INFO:teuthology.orchestra.run.smithi119.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 28 MB/s | 34 MB 00:01 2024-03-29T11:06:31.722 INFO:teuthology.orchestra.run.smithi119.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 19 MB/s | 28 MB 00:01 2024-03-29T11:06:31.725 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:06:31.725 INFO:teuthology.orchestra.run.smithi119.stdout:Total 48 MB/s | 76 MB 00:01 2024-03-29T11:06:31.725 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-03-29T11:06:32.103 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-03-29T11:06:32.103 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-03-29T11:06:32.425 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-03-29T11:06:32.949 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-03-29T11:06:32.949 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-03-29T11:06:33.004 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:06:33.090 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:06:35.365 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-03-29T11:06:35.851 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:06:35.959 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:06:35.999 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:06:36.049 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:06:38.806 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:06:38.943 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:06:40.719 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:06:40.839 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:06:43.706 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:06:43.826 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:06:45.637 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:06:48.584 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:07:14.864 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:07:14.864 INFO:teuthology.orchestra.run.smithi104.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:07:14.864 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:07:14.908 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:07:15.294 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:07:15.294 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:07:15.294 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:07:15.294 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:07:15.697 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:07:15.697 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:07:15.697 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-03-29T11:07:15.697 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:07:15.697 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:07:15.698 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:07:15.698 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:07:15.698 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:07:15.698 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-03-29T11:07:16.016 DEBUG:teuthology.orchestra.run.smithi104:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:07:16.110 INFO:teuthology.orchestra.run.smithi104.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:07:16.111 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:07:16.111 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:07:16.111 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:07:16.111 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:07:16.111 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T11:07:16.111 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi104.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:07:16.112 DEBUG:teuthology.orchestra.run.smithi104:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:07:16.191 INFO:teuthology.orchestra.run.smithi104.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:07:16.191 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:07:16.191 DEBUG:teuthology.orchestra.run.smithi104:> sudo rpm -qi grub2-tools 2024-03-29T11:07:16.225 INFO:teuthology.orchestra.run.smithi104.stdout:Name : grub2-tools 2024-03-29T11:07:16.225 INFO:teuthology.orchestra.run.smithi104.stdout:Epoch : 1 2024-03-29T11:07:16.225 INFO:teuthology.orchestra.run.smithi104.stdout:Version : 2.06 2024-03-29T11:07:16.225 INFO:teuthology.orchestra.run.smithi104.stdout:Release : 68.el9 2024-03-29T11:07:16.226 INFO:teuthology.orchestra.run.smithi104.stdout:Architecture: x86_64 2024-03-29T11:07:16.226 INFO:teuthology.orchestra.run.smithi104.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:07:16.226 INFO:teuthology.orchestra.run.smithi104.stdout:Group : Unspecified 2024-03-29T11:07:16.226 INFO:teuthology.orchestra.run.smithi104.stdout:Size : 8264254 2024-03-29T11:07:16.226 INFO:teuthology.orchestra.run.smithi104.stdout:License : GPLv3+ 2024-03-29T11:07:16.226 INFO:teuthology.orchestra.run.smithi104.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:07:16.226 INFO:teuthology.orchestra.run.smithi104.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:07:16.226 INFO:teuthology.orchestra.run.smithi104.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:07:16.226 INFO:teuthology.orchestra.run.smithi104.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:07:16.226 INFO:teuthology.orchestra.run.smithi104.stdout:Packager : builder@centos.org 2024-03-29T11:07:16.226 INFO:teuthology.orchestra.run.smithi104.stdout:Vendor : CentOS 2024-03-29T11:07:16.226 INFO:teuthology.orchestra.run.smithi104.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:07:16.226 INFO:teuthology.orchestra.run.smithi104.stdout:Summary : Support tools for GRUB. 2024-03-29T11:07:16.226 INFO:teuthology.orchestra.run.smithi104.stdout:Description : 2024-03-29T11:07:16.227 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:07:16.227 INFO:teuthology.orchestra.run.smithi104.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:07:16.227 INFO:teuthology.orchestra.run.smithi104.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:07:16.227 INFO:teuthology.orchestra.run.smithi104.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:07:16.227 INFO:teuthology.orchestra.run.smithi104.stdout:hardware devices. 2024-03-29T11:07:16.227 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:07:16.227 INFO:teuthology.orchestra.run.smithi104.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:07:16.228 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:07:16.228 INFO:teuthology.task.kernel:Updating grub on smithi104 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:07:16.228 DEBUG:teuthology.orchestra.run.smithi104:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:07:16.841 INFO:teuthology.orchestra.run.smithi104.stderr:Generating grub configuration file ... 2024-03-29T11:07:17.825 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:07:17.825 INFO:teuthology.orchestra.run.smithi119.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:07:17.825 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:07:17.870 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:07:18.242 INFO:teuthology.orchestra.run.smithi104.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:07:18.250 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:07:18.251 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:07:18.251 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:07:18.251 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:07:18.293 INFO:teuthology.orchestra.run.smithi104.stderr:done 2024-03-29T11:07:18.296 DEBUG:teuthology.orchestra.run.smithi104:> mktemp 2024-03-29T11:07:18.312 INFO:teuthology.orchestra.run.smithi104.stdout:/tmp/tmp.x63mKUN8Sh 2024-03-29T11:07:18.312 DEBUG:teuthology.orchestra.run.smithi104:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.x63mKUN8Sh 2024-03-29T11:07:18.381 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 0666 /tmp/tmp.x63mKUN8Sh 2024-03-29T11:07:18.518 DEBUG:teuthology.orchestra.remote:smithi104:/tmp/tmp.x63mKUN8Sh is 6KB 2024-03-29T11:07:18.566 DEBUG:teuthology.orchestra.run.smithi104:> rm -fr /tmp/tmp.x63mKUN8Sh 2024-03-29T11:07:18.580 DEBUG:teuthology.orchestra.run.smithi104:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:07:18.644 INFO:teuthology.orchestra.run.smithi104.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:07:18.644 INFO:teuthology.orchestra.run.smithi104.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:07:18.644 INFO:teuthology.orchestra.run.smithi104.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:07:18.644 INFO:teuthology.orchestra.run.smithi104.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:07:18.645 DEBUG:teuthology.orchestra.run.smithi104:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:07:18.651 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:07:18.651 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:07:18.651 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-03-29T11:07:18.651 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:07:18.652 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:07:18.652 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:07:18.652 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:07:18.652 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:07:18.652 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-29T11:07:18.802 DEBUG:teuthology.orchestra.run.smithi104:> sudo shutdown -r now 2024-03-29T11:07:18.978 DEBUG:teuthology.orchestra.run.smithi119:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:07:19.084 INFO:teuthology.orchestra.run.smithi119.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:07:19.084 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:07:19.084 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:07:19.084 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:07:19.084 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:07:19.084 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-29T11:07:19.084 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi119.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:07:19.085 DEBUG:teuthology.orchestra.run.smithi119:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:07:19.165 INFO:teuthology.orchestra.run.smithi119.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:07:19.165 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:07:19.165 DEBUG:teuthology.orchestra.run.smithi119:> sudo rpm -qi grub2-tools 2024-03-29T11:07:19.209 INFO:teuthology.orchestra.run.smithi119.stdout:Name : grub2-tools 2024-03-29T11:07:19.209 INFO:teuthology.orchestra.run.smithi119.stdout:Epoch : 1 2024-03-29T11:07:19.209 INFO:teuthology.orchestra.run.smithi119.stdout:Version : 2.06 2024-03-29T11:07:19.209 INFO:teuthology.orchestra.run.smithi119.stdout:Release : 68.el9 2024-03-29T11:07:19.209 INFO:teuthology.orchestra.run.smithi119.stdout:Architecture: x86_64 2024-03-29T11:07:19.209 INFO:teuthology.orchestra.run.smithi119.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:07:19.209 INFO:teuthology.orchestra.run.smithi119.stdout:Group : Unspecified 2024-03-29T11:07:19.209 INFO:teuthology.orchestra.run.smithi119.stdout:Size : 8264254 2024-03-29T11:07:19.209 INFO:teuthology.orchestra.run.smithi119.stdout:License : GPLv3+ 2024-03-29T11:07:19.209 INFO:teuthology.orchestra.run.smithi119.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:07:19.210 INFO:teuthology.orchestra.run.smithi119.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:07:19.210 INFO:teuthology.orchestra.run.smithi119.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:07:19.210 INFO:teuthology.orchestra.run.smithi119.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:07:19.210 INFO:teuthology.orchestra.run.smithi119.stdout:Packager : builder@centos.org 2024-03-29T11:07:19.210 INFO:teuthology.orchestra.run.smithi119.stdout:Vendor : CentOS 2024-03-29T11:07:19.210 INFO:teuthology.orchestra.run.smithi119.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:07:19.210 INFO:teuthology.orchestra.run.smithi119.stdout:Summary : Support tools for GRUB. 2024-03-29T11:07:19.210 INFO:teuthology.orchestra.run.smithi119.stdout:Description : 2024-03-29T11:07:19.210 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:07:19.210 INFO:teuthology.orchestra.run.smithi119.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:07:19.210 INFO:teuthology.orchestra.run.smithi119.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:07:19.210 INFO:teuthology.orchestra.run.smithi119.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:07:19.210 INFO:teuthology.orchestra.run.smithi119.stdout:hardware devices. 2024-03-29T11:07:19.210 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:07:19.210 INFO:teuthology.orchestra.run.smithi119.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:07:19.211 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:07:19.211 INFO:teuthology.task.kernel:Updating grub on smithi119 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:07:19.212 DEBUG:teuthology.orchestra.run.smithi119:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:07:19.797 INFO:teuthology.orchestra.run.smithi119.stderr:Generating grub configuration file ... 2024-03-29T11:07:21.186 INFO:teuthology.orchestra.run.smithi119.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:07:21.224 INFO:teuthology.orchestra.run.smithi119.stderr:done 2024-03-29T11:07:21.227 DEBUG:teuthology.orchestra.run.smithi119:> mktemp 2024-03-29T11:07:21.241 INFO:teuthology.orchestra.run.smithi119.stdout:/tmp/tmp.AWnRglNVxa 2024-03-29T11:07:21.241 DEBUG:teuthology.orchestra.run.smithi119:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.AWnRglNVxa 2024-03-29T11:07:21.310 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 0666 /tmp/tmp.AWnRglNVxa 2024-03-29T11:07:21.442 DEBUG:teuthology.orchestra.remote:smithi119:/tmp/tmp.AWnRglNVxa is 6KB 2024-03-29T11:07:21.490 DEBUG:teuthology.orchestra.run.smithi119:> rm -fr /tmp/tmp.AWnRglNVxa 2024-03-29T11:07:21.504 DEBUG:teuthology.orchestra.run.smithi119:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:07:21.569 INFO:teuthology.orchestra.run.smithi119.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:07:21.569 INFO:teuthology.orchestra.run.smithi119.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:07:21.569 INFO:teuthology.orchestra.run.smithi119.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:07:21.569 INFO:teuthology.orchestra.run.smithi119.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:07:21.571 DEBUG:teuthology.orchestra.run.smithi119:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:07:21.725 DEBUG:teuthology.orchestra.run.smithi119:> sudo shutdown -r now 2024-03-29T11:07:48.832 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T11:07:48.832 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-03-29T11:07:48.833 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:07:51.730 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-29T11:07:51.730 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-03-29T11:07:51.731 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:08:07.326 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-03-29T11:08:10.142 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.119 2024-03-29T11:08:16.334 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-03-29T11:08:16.335 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:08:19.143 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-03-29T11:08:19.144 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:08:19.390 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-03-29T11:08:22.210 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.119 2024-03-29T11:08:31.398 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-03-29T11:08:31.399 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:08:34.212 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-03-29T11:08:34.213 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:08:52.642 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.119 2024-03-29T11:09:03.354 DEBUG:teuthology.orchestra.run.smithi104:> true 2024-03-29T11:09:03.886 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-03-29T11:09:03.886 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:09:03.886 DEBUG:teuthology.orchestra.run.smithi104:> uname -r 2024-03-29T11:09:03.901 INFO:teuthology.orchestra.run.smithi104.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:09:03.902 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:09:03.902 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:09:03.902 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:09:04.902 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T11:09:04.902 DEBUG:teuthology.orchestra.run.smithi104:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:09:05.013 INFO:teuthology.orchestra.run.smithi104.stdout:ttyS1 2024-03-29T11:09:05.047 DEBUG:teuthology.parallel:result is None 2024-03-29T11:09:07.643 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-03-29T11:09:07.644 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:09:07.955 DEBUG:teuthology.orchestra.run.smithi119:> true 2024-03-29T11:09:08.362 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-03-29T11:09:08.362 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:09:08.363 DEBUG:teuthology.orchestra.run.smithi119:> uname -r 2024-03-29T11:09:08.377 INFO:teuthology.orchestra.run.smithi119.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:09:08.377 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:09:08.377 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:09:08.377 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:09:09.378 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-29T11:09:09.378 DEBUG:teuthology.orchestra.run.smithi119:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:09:09.444 INFO:teuthology.orchestra.run.smithi119.stdout:ttyS1 2024-03-29T11:09:09.498 DEBUG:teuthology.parallel:result is None 2024-03-29T11:09:09.498 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T11:09:09.506 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T11:09:09.506 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:09:09.509 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:09:09.525 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T11:09:09.531 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T11:09:09.542 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T11:09:09.542 DEBUG:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:09:09.566 DEBUG:teuthology.orchestra.run.smithi119:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:09:09.672 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T11:09:09.679 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T11:09:09.679 DEBUG:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:09:09.682 DEBUG:teuthology.orchestra.run.smithi119:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:09:09.720 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:09:09.722 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:09:09.734 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:09:09.735 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:09:09.737 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T11:09:09.744 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T11:09:09.744 DEBUG:teuthology.orchestra.run.smithi104:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:09:09.779 DEBUG:teuthology.orchestra.run.smithi119:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:09:09.806 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T11:09:09.815 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T11:09:09.815 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:09:09.848 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:09:09.863 DEBUG:teuthology.orchestra.run.smithi104:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:09:09.947 DEBUG:teuthology.orchestra.run.smithi104:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:09:10.001 DEBUG:teuthology.orchestra.run.smithi104:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:09:10.084 DEBUG:teuthology.orchestra.run.smithi104:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:09:10.129 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-29T11:09:10.129 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:09:10.197 DEBUG:teuthology.orchestra.run.smithi119:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:09:10.239 DEBUG:teuthology.orchestra.run.smithi119:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:09:10.283 DEBUG:teuthology.orchestra.run.smithi119:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:09:10.361 DEBUG:teuthology.orchestra.run.smithi119:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:09:10.399 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-29T11:09:10.399 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:09:10.462 DEBUG:teuthology.orchestra.run.smithi104:> sudo service rsyslog restart 2024-03-29T11:09:10.464 DEBUG:teuthology.orchestra.run.smithi119:> sudo service rsyslog restart 2024-03-29T11:09:10.514 INFO:teuthology.orchestra.run.smithi104.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:09:10.652 INFO:teuthology.orchestra.run.smithi119.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:09:11.023 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T11:09:11.031 INFO:teuthology.task.internal:Starting timer... 2024-03-29T11:09:11.031 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T11:09:11.086 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T11:09:11.096 DEBUG:teuthology.orchestra.run.smithi104:> sudo service auditd rotate 2024-03-29T11:09:11.149 INFO:teuthology.orchestra.run.smithi104.stdout:Rotating logs: 2024-03-29T11:09:11.151 DEBUG:teuthology.orchestra.run.smithi119:> sudo service auditd rotate 2024-03-29T11:09:11.227 INFO:teuthology.orchestra.run.smithi119.stdout:Rotating logs: 2024-03-29T11:09:11.229 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T11:09:11.229 DEBUG:teuthology.orchestra.run.smithi104:> /usr/sbin/getenforce 2024-03-29T11:09:11.254 INFO:teuthology.orchestra.run.smithi104.stdout:Permissive 2024-03-29T11:09:11.254 DEBUG:teuthology.orchestra.run.smithi119:> /usr/sbin/getenforce 2024-03-29T11:09:11.283 INFO:teuthology.orchestra.run.smithi119.stdout:Permissive 2024-03-29T11:09:11.283 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi104.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi119.front.sepia.ceph.com': 'permissive'} 2024-03-29T11:09:11.283 DEBUG:teuthology.orchestra.run.smithi104:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:09:11.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:09:11.323 DEBUG:teuthology.orchestra.run.smithi119:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:09:11.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:09:11.351 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T11:09:11.351 DEBUG:teuthology.orchestra.run.smithi104:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:09:11.405 DEBUG:teuthology.orchestra.run.smithi119:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:09:11.443 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T11:09:11.453 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-29T11:09:11.458 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-29T11:09:11.575 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T11:09:11.592 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T11:09:11.594 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi104.front.sepia.ceph.com,smithi119.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T11:13:31.666 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi104.front.sepia.ceph.com'), Remote(name='ubuntu@smithi119.front.sepia.ceph.com')] 2024-03-29T11:13:31.668 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-03-29T11:13:31.669 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:13:31.763 DEBUG:teuthology.orchestra.run.smithi104:> true 2024-03-29T11:13:31.840 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-03-29T11:13:31.841 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-03-29T11:13:31.841 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:13:31.928 DEBUG:teuthology.orchestra.run.smithi119:> true 2024-03-29T11:13:32.005 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-03-29T11:13:32.005 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T11:13:32.016 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T11:13:32.016 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:13:32.017 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:13:32.019 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:13:32.019 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:13:32.047 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:13:32.064 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:13:32.077 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:13:32.093 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: ntpd: command not found 2024-03-29T11:13:32.094 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:13:32.106 INFO:teuthology.orchestra.run.smithi104.stdout:506 Cannot talk to daemon 2024-03-29T11:13:32.124 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:13:32.124 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: ntpd: command not found 2024-03-29T11:13:32.137 INFO:teuthology.orchestra.run.smithi119.stdout:506 Cannot talk to daemon 2024-03-29T11:13:32.139 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:13:32.153 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:13:32.169 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:13:32.196 INFO:teuthology.orchestra.run.smithi104.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:13:32.200 INFO:teuthology.orchestra.run.smithi104.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:13:32.200 INFO:teuthology.orchestra.run.smithi104.stdout:=============================================================================== 2024-03-29T11:13:32.200 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:13:32.200 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:13:32.200 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:13:32.200 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:13:32.221 INFO:teuthology.orchestra.run.smithi119.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:13:32.223 INFO:teuthology.orchestra.run.smithi119.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:13:32.224 INFO:teuthology.orchestra.run.smithi119.stdout:=============================================================================== 2024-03-29T11:13:32.224 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:13:32.224 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:13:32.224 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:13:32.224 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:13:32.225 INFO:teuthology.run_tasks:Running task install... 2024-03-29T11:13:32.234 DEBUG:teuthology.task.install:project ceph 2024-03-29T11:13:32.234 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-29T11:13:32.234 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-29T11:13:32.234 INFO:teuthology.task.install:Using flavor: default 2024-03-29T11:13:32.241 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T11:13:32.241 INFO:teuthology.task.install:extra packages: [] 2024-03-29T11:13:32.241 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:13:32.242 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:13:32.243 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:13:32.244 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:13:32.463 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:13:32.464 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:13:32.468 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:13:32.468 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:13:32.579 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:13:32.579 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-29T11:13:32.579 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:13:32.582 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:13:32.583 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-29T11:13:32.583 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:13:32.611 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:13:32.612 DEBUG:teuthology.orchestra.run.smithi104:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:13:32.613 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:13:32.613 DEBUG:teuthology.orchestra.run.smithi119:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:13:32.679 DEBUG:teuthology.orchestra.run.smithi104:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:13:32.680 DEBUG:teuthology.orchestra.run.smithi119:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:13:32.760 DEBUG:teuthology.orchestra.run.smithi104:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:13:32.761 DEBUG:teuthology.orchestra.run.smithi119:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:13:32.826 INFO:teuthology.orchestra.run.smithi104.stdout:check_obsoletes = 1 2024-03-29T11:13:32.827 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum clean all 2024-03-29T11:13:32.829 INFO:teuthology.orchestra.run.smithi119.stdout:check_obsoletes = 1 2024-03-29T11:13:32.830 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum clean all 2024-03-29T11:13:33.067 INFO:teuthology.orchestra.run.smithi119.stdout:74 files removed 2024-03-29T11:13:33.069 INFO:teuthology.orchestra.run.smithi104.stdout:74 files removed 2024-03-29T11:13:33.094 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-radosgw 2024-03-29T11:13:33.095 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-radosgw 2024-03-29T11:13:33.704 INFO:teuthology.orchestra.run.smithi104.stdout:ceph packages for x86_64 264 kB/s | 82 kB 00:00 2024-03-29T11:13:33.799 INFO:teuthology.orchestra.run.smithi119.stdout:ceph packages for x86_64 203 kB/s | 82 kB 00:00 2024-03-29T11:13:34.146 INFO:teuthology.orchestra.run.smithi104.stdout:ceph noarch packages 26 kB/s | 10 kB 00:00 2024-03-29T11:13:34.224 INFO:teuthology.orchestra.run.smithi119.stdout:ceph noarch packages 27 kB/s | 10 kB 00:00 2024-03-29T11:13:34.564 INFO:teuthology.orchestra.run.smithi104.stdout:ceph source packages 5.0 kB/s | 1.9 kB 00:00 2024-03-29T11:13:34.575 INFO:teuthology.orchestra.run.smithi119.stdout:ceph source packages 6.1 kB/s | 1.9 kB 00:00 2024-03-29T11:13:35.000 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-29T11:13:35.061 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-29T11:13:36.461 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS-9 - BaseOS 7.5 MB/s | 2.0 MB 00:00 2024-03-29T11:13:36.543 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS-9 - BaseOS 4.7 MB/s | 2.0 MB 00:00 2024-03-29T11:13:37.484 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-03-29T11:13:37.521 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-29T11:13:39.993 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - AppStream 13 MB/s | 19 MB 00:01 2024-03-29T11:13:44.311 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - CRB 7.5 MB/s | 6.1 MB 00:00 2024-03-29T11:13:45.919 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 16 kB 00:00 2024-03-29T11:13:47.440 INFO:teuthology.orchestra.run.smithi104.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-03-29T11:13:52.453 INFO:teuthology.orchestra.run.smithi104.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-29T11:13:55.066 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-03-29T11:13:55.069 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-03-29T11:13:55.069 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-03-29T11:13:55.069 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-03-29T11:13:55.069 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-03-29T11:13:55.069 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:13:55.070 INFO:teuthology.orchestra.run.smithi104.stdout:Upgrading: 2024-03-29T11:13:55.070 INFO:teuthology.orchestra.run.smithi104.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:13:55.070 INFO:teuthology.orchestra.run.smithi104.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:13:55.070 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-03-29T11:13:55.070 INFO:teuthology.orchestra.run.smithi104.stdout: boost-program-options 2024-03-29T11:13:55.070 INFO:teuthology.orchestra.run.smithi104.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:13:55.070 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:13:55.070 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:13:55.070 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:13:55.070 INFO:teuthology.orchestra.run.smithi104.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:13:55.070 INFO:teuthology.orchestra.run.smithi104.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:13:55.070 INFO:teuthology.orchestra.run.smithi104.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:13:55.070 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:13:55.071 INFO:teuthology.orchestra.run.smithi104.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:13:55.071 INFO:teuthology.orchestra.run.smithi104.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:13:55.071 INFO:teuthology.orchestra.run.smithi104.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:13:55.071 INFO:teuthology.orchestra.run.smithi104.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:13:55.071 INFO:teuthology.orchestra.run.smithi104.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:13:55.071 INFO:teuthology.orchestra.run.smithi104.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:13:55.071 INFO:teuthology.orchestra.run.smithi104.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:13:55.071 INFO:teuthology.orchestra.run.smithi104.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:13:55.071 INFO:teuthology.orchestra.run.smithi104.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:13:55.071 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-argparse 2024-03-29T11:13:55.071 INFO:teuthology.orchestra.run.smithi104.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:13:55.071 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:13:55.071 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:13:55.071 INFO:teuthology.orchestra.run.smithi104.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:13:55.072 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:13:55.072 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:13:55.072 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:13:55.072 INFO:teuthology.orchestra.run.smithi104.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:13:55.072 INFO:teuthology.orchestra.run.smithi104.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:13:55.072 INFO:teuthology.orchestra.run.smithi104.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:13:55.072 INFO:teuthology.orchestra.run.smithi104.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:13:55.072 INFO:teuthology.orchestra.run.smithi104.stdout:Installing weak dependencies: 2024-03-29T11:13:55.072 INFO:teuthology.orchestra.run.smithi104.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:13:55.072 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:13:55.072 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-03-29T11:13:55.072 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-03-29T11:13:55.072 INFO:teuthology.orchestra.run.smithi104.stdout:Install 30 Packages 2024-03-29T11:13:55.072 INFO:teuthology.orchestra.run.smithi104.stdout:Upgrade 2 Packages 2024-03-29T11:13:55.072 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:13:55.073 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 58 M 2024-03-29T11:13:55.073 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-03-29T11:13:56.021 INFO:teuthology.orchestra.run.smithi104.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 9.5 MB/s | 5.3 MB 00:00 2024-03-29T11:13:56.079 INFO:teuthology.orchestra.run.smithi104.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 434 kB/s | 25 kB 00:00 2024-03-29T11:13:56.172 INFO:teuthology.orchestra.run.smithi104.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 7.9 MB/s | 742 kB 00:00 2024-03-29T11:13:56.248 INFO:teuthology.orchestra.run.smithi104.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 6.5 MB/s | 498 kB 00:00 2024-03-29T11:13:56.597 INFO:teuthology.orchestra.run.smithi104.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 15 MB/s | 5.1 MB 00:00 2024-03-29T11:13:56.639 INFO:teuthology.orchestra.run.smithi104.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 1.1 MB/s | 45 kB 00:00 2024-03-29T11:13:56.697 INFO:teuthology.orchestra.run.smithi104.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 2.2 MB/s | 131 kB 00:00 2024-03-29T11:13:56.798 INFO:teuthology.orchestra.run.smithi104.stdout:(8/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 7.3 MB/s | 9.7 MB 00:01 2024-03-29T11:13:56.823 INFO:teuthology.orchestra.run.smithi104.stdout:(9/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 1.3 MB/s | 162 kB 00:00 2024-03-29T11:13:56.848 INFO:teuthology.orchestra.run.smithi104.stdout:(10/32): python3-rados-19.0.0-1439.g5dcb190d.el 6.3 MB/s | 320 kB 00:00 2024-03-29T11:13:56.882 INFO:teuthology.orchestra.run.smithi104.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 5.0 MB/s | 298 kB 00:00 2024-03-29T11:13:56.907 INFO:teuthology.orchestra.run.smithi104.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.7 MB/s | 99 kB 00:00 2024-03-29T11:13:57.024 INFO:teuthology.orchestra.run.smithi104.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 747 kB/s | 106 kB 00:00 2024-03-29T11:13:57.049 INFO:teuthology.orchestra.run.smithi104.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 159 kB 00:00 2024-03-29T11:13:57.074 INFO:teuthology.orchestra.run.smithi104.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 884 kB/s | 44 kB 00:00 2024-03-29T11:13:57.125 INFO:teuthology.orchestra.run.smithi104.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-03-29T11:13:57.150 INFO:teuthology.orchestra.run.smithi104.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-29T11:13:57.175 INFO:teuthology.orchestra.run.smithi104.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 825 kB/s | 41 kB 00:00 2024-03-29T11:13:57.201 INFO:teuthology.orchestra.run.smithi104.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2024-03-29T11:13:57.226 INFO:teuthology.orchestra.run.smithi104.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 698 kB/s | 35 kB 00:00 2024-03-29T11:13:57.251 INFO:teuthology.orchestra.run.smithi104.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 923 kB/s | 46 kB 00:00 2024-03-29T11:13:57.460 INFO:teuthology.orchestra.run.smithi104.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.3 MB/s | 309 kB 00:00 2024-03-29T11:13:57.510 INFO:teuthology.orchestra.run.smithi104.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 524 kB/s | 26 kB 00:00 2024-03-29T11:13:57.568 INFO:teuthology.orchestra.run.smithi104.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 836 kB/s | 49 kB 00:00 2024-03-29T11:13:57.627 INFO:teuthology.orchestra.run.smithi104.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-29T11:13:57.811 INFO:teuthology.orchestra.run.smithi104.stdout:(26/32): libarrow-9.0.0-10.el9.x86_64.rpm 7.9 MB/s | 4.4 MB 00:00 2024-03-29T11:13:57.836 INFO:teuthology.orchestra.run.smithi104.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.9 MB/s | 838 kB 00:00 2024-03-29T11:13:57.870 INFO:teuthology.orchestra.run.smithi104.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-03-29T11:13:58.004 INFO:teuthology.orchestra.run.smithi104.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 9.5 MB/s | 1.6 MB 00:00 2024-03-29T11:13:58.170 INFO:teuthology.orchestra.run.smithi104.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 11 MB/s | 3.4 MB 00:00 2024-03-29T11:13:58.354 INFO:teuthology.orchestra.run.smithi104.stdout:(31/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 6.9 MB/s | 20 MB 00:02 2024-03-29T11:13:58.488 INFO:teuthology.orchestra.run.smithi104.stdout:(32/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 6.5 MB/s | 3.1 MB 00:00 2024-03-29T11:13:58.493 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:13:58.494 INFO:teuthology.orchestra.run.smithi104.stdout:Total 17 MB/s | 58 MB 00:03 2024-03-29T11:13:58.494 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-03-29T11:13:58.528 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-03-29T11:13:58.529 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-03-29T11:13:58.994 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-03-29T11:13:58.994 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-03-29T11:14:00.067 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - AppStream 902 kB/s | 19 MB 00:21 2024-03-29T11:14:00.439 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-03-29T11:14:00.529 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:14:00.567 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:14:00.593 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:14:00.624 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:14:00.640 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:14:00.835 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:14:00.888 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:14:01.274 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:14:01.302 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:14:01.377 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:14:01.467 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:14:01.497 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:14:01.542 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:14:01.570 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:14:01.614 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:14:01.674 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:14:01.698 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:14:01.762 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:14:01.796 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:14:01.905 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:14:01.939 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:14:01.979 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:14:02.012 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:14:02.041 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:14:02.074 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:14:02.397 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:14:02.421 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:14:02.510 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:14:02.542 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:14:02.585 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:14:03.041 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:14:03.073 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:14:03.128 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:14:03.196 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:14:03.225 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:14:04.559 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - CRB 6.1 MB/s | 6.1 MB 00:01 2024-03-29T11:14:05.364 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:14:05.413 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:14:05.956 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:14:05.991 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:14:06.011 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - Extras packages 57 kB/s | 16 kB 00:00 2024-03-29T11:14:06.075 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:14:06.075 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:14:06.075 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:14:06.075 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:14:06.117 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:14:09.095 INFO:teuthology.orchestra.run.smithi119.stdout:Extra Packages for Enterprise Linux 7.0 MB/s | 21 MB 00:02 2024-03-29T11:14:14.128 INFO:teuthology.orchestra.run.smithi119.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-29T11:14:15.700 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:14:15.700 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /sys 2024-03-29T11:14:15.700 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /proc 2024-03-29T11:14:15.700 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /mnt 2024-03-29T11:14:15.700 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /var/tmp 2024-03-29T11:14:15.700 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /home 2024-03-29T11:14:15.700 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /root 2024-03-29T11:14:15.700 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /tmp 2024-03-29T11:14:15.700 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:14:16.620 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:14:16.645 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:14:16.645 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:14:16.645 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:14:16.645 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:14:16.645 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:14:16.645 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:14:16.647 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:14:16.723 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:14:16.723 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:14:16.760 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-29T11:14:16.767 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-29T11:14:16.768 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-03-29T11:14:16.768 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-29T11:14:16.768 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-03-29T11:14:16.768 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:14:16.768 INFO:teuthology.orchestra.run.smithi119.stdout:Upgrading: 2024-03-29T11:14:16.768 INFO:teuthology.orchestra.run.smithi119.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:14:16.768 INFO:teuthology.orchestra.run.smithi119.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:14:16.768 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-03-29T11:14:16.768 INFO:teuthology.orchestra.run.smithi119.stdout: boost-program-options 2024-03-29T11:14:16.768 INFO:teuthology.orchestra.run.smithi119.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:14:16.768 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:14:16.768 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:14:16.768 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:14:16.768 INFO:teuthology.orchestra.run.smithi119.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:14:16.769 INFO:teuthology.orchestra.run.smithi119.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:14:16.769 INFO:teuthology.orchestra.run.smithi119.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:14:16.769 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:14:16.769 INFO:teuthology.orchestra.run.smithi119.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:14:16.769 INFO:teuthology.orchestra.run.smithi119.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:14:16.769 INFO:teuthology.orchestra.run.smithi119.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:14:16.769 INFO:teuthology.orchestra.run.smithi119.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:14:16.769 INFO:teuthology.orchestra.run.smithi119.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:14:16.769 INFO:teuthology.orchestra.run.smithi119.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:14:16.769 INFO:teuthology.orchestra.run.smithi119.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:14:16.769 INFO:teuthology.orchestra.run.smithi119.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:14:16.769 INFO:teuthology.orchestra.run.smithi119.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:14:16.769 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-argparse 2024-03-29T11:14:16.769 INFO:teuthology.orchestra.run.smithi119.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:14:16.770 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:14:16.770 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:14:16.770 INFO:teuthology.orchestra.run.smithi119.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:14:16.770 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:14:16.770 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:14:16.770 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:14:16.770 INFO:teuthology.orchestra.run.smithi119.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:14:16.770 INFO:teuthology.orchestra.run.smithi119.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:14:16.770 INFO:teuthology.orchestra.run.smithi119.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:14:16.770 INFO:teuthology.orchestra.run.smithi119.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:14:16.770 INFO:teuthology.orchestra.run.smithi119.stdout:Installing weak dependencies: 2024-03-29T11:14:16.770 INFO:teuthology.orchestra.run.smithi119.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:14:16.770 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:14:16.770 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-03-29T11:14:16.770 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-29T11:14:16.771 INFO:teuthology.orchestra.run.smithi119.stdout:Install 30 Packages 2024-03-29T11:14:16.771 INFO:teuthology.orchestra.run.smithi119.stdout:Upgrade 2 Packages 2024-03-29T11:14:16.771 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:14:16.771 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 58 M 2024-03-29T11:14:16.771 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-03-29T11:14:17.804 INFO:teuthology.orchestra.run.smithi119.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 5.4 MB/s | 5.3 MB 00:00 2024-03-29T11:14:17.849 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:14:17.850 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:14:17.850 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:14:17.850 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:14:17.850 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:14:17.850 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:14:17.850 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:14:17.850 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:14:17.850 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:14:17.850 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:14:17.850 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:14:17.850 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:14:17.850 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:14:17.851 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:14:17.851 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:14:17.851 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:14:17.851 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:14:17.851 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:14:17.851 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:14:17.851 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:14:17.851 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:14:17.851 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:14:17.853 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:14:17.854 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:14:17.854 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:14:17.854 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:14:17.854 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:14:17.854 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:14:17.855 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:14:17.855 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:14:17.855 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:14:17.855 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:14:17.855 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:14:17.855 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:14:17.938 INFO:teuthology.orchestra.run.smithi119.stdout:(2/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 8.7 MB/s | 9.7 MB 00:01 2024-03-29T11:14:17.954 INFO:teuthology.orchestra.run.smithi119.stdout:(3/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 167 kB/s | 25 kB 00:00 2024-03-29T11:14:18.071 INFO:teuthology.orchestra.run.smithi119.stdout:(4/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 5.4 MB/s | 742 kB 00:00 2024-03-29T11:14:18.105 INFO:teuthology.orchestra.run.smithi119.stdout:(5/32): libradosstriper1-19.0.0-1439.g5dcb190d. 3.2 MB/s | 498 kB 00:00 2024-03-29T11:14:18.172 INFO:teuthology.orchestra.run.smithi119.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 677 kB/s | 45 kB 00:00 2024-03-29T11:14:18.247 INFO:teuthology.orchestra.run.smithi119.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.7 MB/s | 131 kB 00:00 2024-03-29T11:14:18.323 INFO:teuthology.orchestra.run.smithi119.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 2.1 MB/s | 162 kB 00:00 2024-03-29T11:14:18.407 INFO:teuthology.orchestra.run.smithi119.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 3.8 MB/s | 320 kB 00:00 2024-03-29T11:14:18.490 INFO:teuthology.orchestra.run.smithi119.stdout:(10/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 12 MB/s | 5.1 MB 00:00 2024-03-29T11:14:18.515 INFO:teuthology.orchestra.run.smithi119.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 2.7 MB/s | 298 kB 00:00 2024-03-29T11:14:18.549 INFO:teuthology.orchestra.run.smithi119.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.7 MB/s | 99 kB 00:00 2024-03-29T11:14:18.852 INFO:teuthology.orchestra.run.smithi119.stdout:(13/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 9.8 MB/s | 20 MB 00:02 2024-03-29T11:14:18.874 INFO:teuthology.orchestra.run.smithi119.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 295 kB/s | 106 kB 00:00 2024-03-29T11:14:18.900 INFO:teuthology.orchestra.run.smithi119.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 889 kB/s | 44 kB 00:00 2024-03-29T11:14:18.925 INFO:teuthology.orchestra.run.smithi119.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 453 kB/s | 159 kB 00:00 2024-03-29T11:14:18.964 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:14:18.964 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:14:18.964 INFO:teuthology.orchestra.run.smithi104.stdout:Upgraded: 2024-03-29T11:14:18.964 INFO:teuthology.orchestra.run.smithi104.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:14:18.964 INFO:teuthology.orchestra.run.smithi104.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:14:18.964 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-03-29T11:14:18.964 INFO:teuthology.orchestra.run.smithi104.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:14:18.965 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:14:18.965 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:14:18.965 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:14:18.965 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:14:18.965 INFO:teuthology.orchestra.run.smithi104.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:14:18.965 INFO:teuthology.orchestra.run.smithi104.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:14:18.965 INFO:teuthology.orchestra.run.smithi104.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:14:18.965 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:14:18.965 INFO:teuthology.orchestra.run.smithi104.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:14:18.965 INFO:teuthology.orchestra.run.smithi104.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:14:18.965 INFO:teuthology.orchestra.run.smithi104.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:14:18.965 INFO:teuthology.orchestra.run.smithi104.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:14:18.965 INFO:teuthology.orchestra.run.smithi104.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:14:18.965 INFO:teuthology.orchestra.run.smithi104.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:14:18.966 INFO:teuthology.orchestra.run.smithi104.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:14:18.966 INFO:teuthology.orchestra.run.smithi104.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:14:18.966 INFO:teuthology.orchestra.run.smithi104.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:14:18.966 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:14:18.966 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:14:18.966 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:14:18.966 INFO:teuthology.orchestra.run.smithi104.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:14:18.966 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:14:18.966 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:14:18.966 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:14:18.966 INFO:teuthology.orchestra.run.smithi104.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:14:18.966 INFO:teuthology.orchestra.run.smithi104.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:14:18.966 INFO:teuthology.orchestra.run.smithi104.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:14:18.966 INFO:teuthology.orchestra.run.smithi104.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:14:18.966 INFO:teuthology.orchestra.run.smithi104.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:14:18.967 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:14:18.967 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-03-29T11:14:19.009 INFO:teuthology.orchestra.run.smithi119.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.8 MB/s | 661 kB 00:00 2024-03-29T11:14:19.034 INFO:teuthology.orchestra.run.smithi119.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 282 kB 00:00 2024-03-29T11:14:19.050 INFO:teuthology.orchestra.run.smithi119.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 328 kB/s | 41 kB 00:00 2024-03-29T11:14:19.076 INFO:teuthology.orchestra.run.smithi119.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 3.2 MB/s | 220 kB 00:00 2024-03-29T11:14:19.109 INFO:teuthology.orchestra.run.smithi119.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 799 kB/s | 46 kB 00:00 2024-03-29T11:14:19.135 INFO:teuthology.orchestra.run.smithi119.stdout:(22/32): qatlib-service-24.02.0-1.el9.x86_64.rp 417 kB/s | 35 kB 00:00 2024-03-29T11:14:19.267 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-test 2024-03-29T11:14:19.543 INFO:teuthology.orchestra.run.smithi119.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 64 kB/s | 26 kB 00:00 2024-03-29T11:14:19.735 INFO:teuthology.orchestra.run.smithi119.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 253 kB/s | 49 kB 00:00 2024-03-29T11:14:19.760 INFO:teuthology.orchestra.run.smithi119.stdout:(25/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 451 kB/s | 309 kB 00:00 2024-03-29T11:14:19.831 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:00:27 ago on Fri 29 Mar 2024 11:13:52 AM UTC. 2024-03-29T11:14:19.886 INFO:teuthology.orchestra.run.smithi119.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 449 kB/s | 67 kB 00:00 2024-03-29T11:14:19.984 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-03-29T11:14:19.985 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-03-29T11:14:19.985 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-03-29T11:14:19.985 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-03-29T11:14:19.985 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-03-29T11:14:19.985 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:14:19.986 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-03-29T11:14:19.986 INFO:teuthology.orchestra.run.smithi104.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:14:19.986 INFO:teuthology.orchestra.run.smithi104.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:14:19.986 INFO:teuthology.orchestra.run.smithi104.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:14:19.986 INFO:teuthology.orchestra.run.smithi104.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:14:19.986 INFO:teuthology.orchestra.run.smithi104.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:14:19.986 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:14:19.986 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-03-29T11:14:19.986 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-03-29T11:14:19.986 INFO:teuthology.orchestra.run.smithi104.stdout:Install 6 Packages 2024-03-29T11:14:19.986 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:14:19.987 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 48 M 2024-03-29T11:14:19.987 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 204 M 2024-03-29T11:14:19.987 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-03-29T11:14:20.044 INFO:teuthology.orchestra.run.smithi119.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.9 MB/s | 838 kB 00:00 2024-03-29T11:14:20.103 INFO:teuthology.orchestra.run.smithi119.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 883 kB/s | 192 kB 00:00 2024-03-29T11:14:20.227 INFO:teuthology.orchestra.run.smithi104.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-29T11:14:20.253 INFO:teuthology.orchestra.run.smithi119.stdout:(29/32): libarrow-9.0.0-10.el9.x86_64.rpm 3.9 MB/s | 4.4 MB 00:01 2024-03-29T11:14:20.285 INFO:teuthology.orchestra.run.smithi104.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-29T11:14:20.295 INFO:teuthology.orchestra.run.smithi119.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 6.3 MB/s | 1.6 MB 00:00 2024-03-29T11:14:20.319 INFO:teuthology.orchestra.run.smithi104.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-29T11:14:20.494 INFO:teuthology.orchestra.run.smithi104.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 366 kB/s | 163 kB 00:00 2024-03-29T11:14:20.512 INFO:teuthology.orchestra.run.smithi119.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 8.3 MB/s | 3.4 MB 00:00 2024-03-29T11:14:20.597 INFO:teuthology.orchestra.run.smithi119.stdout:(32/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 9.2 MB/s | 3.1 MB 00:00 2024-03-29T11:14:20.603 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:14:20.604 INFO:teuthology.orchestra.run.smithi119.stdout:Total 15 MB/s | 58 MB 00:03 2024-03-29T11:14:20.604 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-03-29T11:14:20.641 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-03-29T11:14:20.642 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-03-29T11:14:20.903 INFO:teuthology.orchestra.run.smithi104.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 321 kB/s | 187 kB 00:00 2024-03-29T11:14:21.125 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-03-29T11:14:21.125 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-03-29T11:14:22.576 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-03-29T11:14:22.647 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:14:22.675 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:14:22.700 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:14:22.723 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:14:22.740 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:14:22.940 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:14:22.995 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:14:23.368 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:14:23.402 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:14:23.494 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:14:23.583 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:14:23.621 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:14:23.675 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:14:23.704 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:14:23.748 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:14:23.817 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:14:23.840 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:14:23.912 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:14:23.946 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:14:24.058 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:14:24.098 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:14:24.138 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:14:24.171 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:14:24.200 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:14:24.233 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:14:24.551 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:14:24.579 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:14:24.668 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:14:24.701 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:14:24.743 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:14:25.200 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:14:25.240 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:14:25.303 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:14:25.370 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:14:25.401 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:14:26.787 INFO:teuthology.orchestra.run.smithi104.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 7.0 MB/s | 47 MB 00:06 2024-03-29T11:14:26.900 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:14:26.900 INFO:teuthology.orchestra.run.smithi104.stdout:Total 7.1 MB/s | 48 MB 00:06 2024-03-29T11:14:26.901 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-03-29T11:14:26.902 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-03-29T11:14:26.902 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-03-29T11:14:27.200 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-03-29T11:14:27.201 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-03-29T11:14:27.540 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:14:27.581 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:14:27.775 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-03-29T11:14:27.853 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:14:27.906 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:14:27.944 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:14:27.999 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:14:28.022 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:14:28.126 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:14:28.159 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:14:28.231 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:14:28.232 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:14:28.232 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:14:28.232 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:14:28.268 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:14:32.489 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:14:32.572 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:14:35.057 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:14:35.058 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:14:35.058 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:14:35.058 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:14:35.058 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:14:35.058 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:14:35.474 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:14:35.475 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:14:35.475 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-03-29T11:14:35.475 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:14:35.475 INFO:teuthology.orchestra.run.smithi104.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:14:35.475 INFO:teuthology.orchestra.run.smithi104.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:14:35.475 INFO:teuthology.orchestra.run.smithi104.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:14:35.475 INFO:teuthology.orchestra.run.smithi104.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:14:35.475 INFO:teuthology.orchestra.run.smithi104.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:14:35.475 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:14:35.475 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-03-29T11:14:35.809 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph 2024-03-29T11:14:36.375 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:00:44 ago on Fri 29 Mar 2024 11:13:52 AM UTC. 2024-03-29T11:14:36.535 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-03-29T11:14:36.537 INFO:teuthology.orchestra.run.smithi104.stdout:========================================================================================= 2024-03-29T11:14:36.537 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-03-29T11:14:36.537 INFO:teuthology.orchestra.run.smithi104.stdout:========================================================================================= 2024-03-29T11:14:36.537 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-03-29T11:14:36.537 INFO:teuthology.orchestra.run.smithi104.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:14:36.537 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-03-29T11:14:36.537 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:14:36.537 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:14:36.537 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:14:36.537 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:14:36.537 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:14:36.537 INFO:teuthology.orchestra.run.smithi104.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:14:36.538 INFO:teuthology.orchestra.run.smithi104.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:14:36.538 INFO:teuthology.orchestra.run.smithi104.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:14:36.538 INFO:teuthology.orchestra.run.smithi104.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:14:36.538 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:14:36.538 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:14:36.538 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:14:36.538 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:14:36.538 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:14:36.538 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:14:36.538 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:14:36.538 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:14:36.538 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:14:36.538 INFO:teuthology.orchestra.run.smithi104.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:14:36.539 INFO:teuthology.orchestra.run.smithi104.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:14:36.539 INFO:teuthology.orchestra.run.smithi104.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:14:36.539 INFO:teuthology.orchestra.run.smithi104.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:14:36.539 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:14:36.539 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:14:36.539 INFO:teuthology.orchestra.run.smithi104.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:14:36.539 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:14:36.539 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:14:36.539 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:14:36.539 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:14:36.539 INFO:teuthology.orchestra.run.smithi104.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:14:36.539 INFO:teuthology.orchestra.run.smithi104.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:14:36.539 INFO:teuthology.orchestra.run.smithi104.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:14:36.539 INFO:teuthology.orchestra.run.smithi104.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:14:36.539 INFO:teuthology.orchestra.run.smithi104.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:14:36.540 INFO:teuthology.orchestra.run.smithi104.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:14:36.540 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:14:36.540 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-03-29T11:14:36.540 INFO:teuthology.orchestra.run.smithi104.stdout:========================================================================================= 2024-03-29T11:14:36.540 INFO:teuthology.orchestra.run.smithi104.stdout:Install 36 Packages 2024-03-29T11:14:36.540 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:14:36.541 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 30 M 2024-03-29T11:14:36.541 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 107 M 2024-03-29T11:14:36.541 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-03-29T11:14:37.105 INFO:teuthology.orchestra.run.smithi104.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 40 kB/s | 6.6 kB 00:00 2024-03-29T11:14:37.355 INFO:teuthology.orchestra.run.smithi104.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 5.1 MB/s | 2.1 MB 00:00 2024-03-29T11:14:37.555 INFO:teuthology.orchestra.run.smithi104.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 2.4 MB/s | 1.5 MB 00:00 2024-03-29T11:14:37.672 INFO:teuthology.orchestra.run.smithi104.stdout:(4/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 8.3 MB/s | 4.7 MB 00:00 2024-03-29T11:14:37.697 INFO:teuthology.orchestra.run.smithi104.stdout:(5/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 1.7 MB/s | 247 kB 00:00 2024-03-29T11:14:37.719 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:14:37.719 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /sys 2024-03-29T11:14:37.719 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /proc 2024-03-29T11:14:37.719 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /mnt 2024-03-29T11:14:37.719 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /var/tmp 2024-03-29T11:14:37.719 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /home 2024-03-29T11:14:37.719 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /root 2024-03-29T11:14:37.720 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /tmp 2024-03-29T11:14:37.720 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:14:37.822 INFO:teuthology.orchestra.run.smithi104.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-29T11:14:37.849 INFO:teuthology.orchestra.run.smithi104.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 961 kB/s | 144 kB 00:00 2024-03-29T11:14:37.865 INFO:teuthology.orchestra.run.smithi104.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 756 kB/s | 32 kB 00:00 2024-03-29T11:14:37.881 INFO:teuthology.orchestra.run.smithi104.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-29T11:14:37.898 INFO:teuthology.orchestra.run.smithi104.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-29T11:14:37.924 INFO:teuthology.orchestra.run.smithi104.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-29T11:14:37.975 INFO:teuthology.orchestra.run.smithi104.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-29T11:14:38.016 INFO:teuthology.orchestra.run.smithi104.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2024-03-29T11:14:38.041 INFO:teuthology.orchestra.run.smithi104.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 1.8 MB/s | 124 kB 00:00 2024-03-29T11:14:38.066 INFO:teuthology.orchestra.run.smithi104.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-29T11:14:38.425 INFO:teuthology.orchestra.run.smithi104.stdout:(16/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 16 MB/s | 17 MB 00:01 2024-03-29T11:14:38.451 INFO:teuthology.orchestra.run.smithi104.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 58 kB/s | 22 kB 00:00 2024-03-29T11:14:38.600 INFO:teuthology.orchestra.run.smithi104.stdout:(18/36): python3-requests-2.25.1-8.el9.noarch.r 223 kB/s | 125 kB 00:00 2024-03-29T11:14:38.626 INFO:teuthology.orchestra.run.smithi104.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 248 kB/s | 43 kB 00:00 2024-03-29T11:14:38.642 INFO:teuthology.orchestra.run.smithi104.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 696 kB/s | 151 kB 00:00 2024-03-29T11:14:38.652 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:14:38.676 INFO:teuthology.orchestra.run.smithi104.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-03-29T11:14:38.677 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:14:38.677 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:14:38.677 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:14:38.677 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:14:38.677 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:14:38.677 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:14:38.679 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:14:38.744 INFO:teuthology.orchestra.run.smithi104.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.0 MB/s | 358 kB 00:00 2024-03-29T11:14:38.764 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:14:38.764 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:14:38.781 INFO:teuthology.orchestra.run.smithi104.stdout:(23/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 77 kB/s | 11 kB 00:00 2024-03-29T11:14:38.796 INFO:teuthology.orchestra.run.smithi104.stdout:(24/36): python3-jaraco-classes-3.2.1-5.el9.noa 148 kB/s | 18 kB 00:00 2024-03-29T11:14:38.813 INFO:teuthology.orchestra.run.smithi104.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 334 kB/s | 23 kB 00:00 2024-03-29T11:14:38.838 INFO:teuthology.orchestra.run.smithi104.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 466 kB/s | 19 kB 00:00 2024-03-29T11:14:38.855 INFO:teuthology.orchestra.run.smithi104.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 335 kB/s | 20 kB 00:00 2024-03-29T11:14:38.871 INFO:teuthology.orchestra.run.smithi104.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 788 kB/s | 46 kB 00:00 2024-03-29T11:14:38.905 INFO:teuthology.orchestra.run.smithi104.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-03-29T11:14:38.947 INFO:teuthology.orchestra.run.smithi104.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.9 MB/s | 272 kB 00:00 2024-03-29T11:14:38.964 INFO:teuthology.orchestra.run.smithi104.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 178 kB/s | 16 kB 00:00 2024-03-29T11:14:38.989 INFO:teuthology.orchestra.run.smithi104.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.1 MB/s | 90 kB 00:00 2024-03-29T11:14:39.006 INFO:teuthology.orchestra.run.smithi104.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 611 kB/s | 36 kB 00:00 2024-03-29T11:14:39.065 INFO:teuthology.orchestra.run.smithi104.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-29T11:14:39.115 INFO:teuthology.orchestra.run.smithi104.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.3 MB/s | 427 kB 00:00 2024-03-29T11:14:39.131 INFO:teuthology.orchestra.run.smithi104.stdout:(36/36): python3-zc-lockfile-2.0-10.el9.noarch. 159 kB/s | 20 kB 00:00 2024-03-29T11:14:39.137 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:14:39.137 INFO:teuthology.orchestra.run.smithi104.stdout:Total 11 MB/s | 30 MB 00:02 2024-03-29T11:14:39.159 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-03-29T11:14:39.197 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-03-29T11:14:39.197 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-03-29T11:14:39.586 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-03-29T11:14:39.587 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-03-29T11:14:39.855 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:14:39.855 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:14:39.855 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:14:39.855 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:14:39.855 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:14:39.855 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:14:39.856 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:14:39.856 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:14:39.856 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:14:39.856 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:14:39.856 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:14:39.856 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:14:39.856 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:14:39.856 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:14:39.856 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:14:39.856 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:14:39.857 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:14:39.857 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:14:39.857 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:14:39.857 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:14:39.857 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:14:39.857 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:14:39.857 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:14:39.857 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:14:39.857 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:14:39.857 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:14:39.858 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:14:39.858 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:14:39.858 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:14:39.858 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:14:39.859 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:14:39.859 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:14:39.859 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:14:39.859 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:14:40.875 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:14:40.875 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:14:40.875 INFO:teuthology.orchestra.run.smithi119.stdout:Upgraded: 2024-03-29T11:14:40.875 INFO:teuthology.orchestra.run.smithi119.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:14:40.875 INFO:teuthology.orchestra.run.smithi119.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:14:40.875 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-03-29T11:14:40.875 INFO:teuthology.orchestra.run.smithi119.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:14:40.876 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:14:40.876 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:14:40.876 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:14:40.876 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:14:40.876 INFO:teuthology.orchestra.run.smithi119.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:14:40.876 INFO:teuthology.orchestra.run.smithi119.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:14:40.876 INFO:teuthology.orchestra.run.smithi119.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:14:40.876 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:14:40.876 INFO:teuthology.orchestra.run.smithi119.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:14:40.876 INFO:teuthology.orchestra.run.smithi119.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:14:40.876 INFO:teuthology.orchestra.run.smithi119.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:14:40.876 INFO:teuthology.orchestra.run.smithi119.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:14:40.876 INFO:teuthology.orchestra.run.smithi119.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:14:40.876 INFO:teuthology.orchestra.run.smithi119.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:14:40.877 INFO:teuthology.orchestra.run.smithi119.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:14:40.877 INFO:teuthology.orchestra.run.smithi119.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:14:40.877 INFO:teuthology.orchestra.run.smithi119.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:14:40.877 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:14:40.877 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:14:40.877 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:14:40.877 INFO:teuthology.orchestra.run.smithi119.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:14:40.877 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:14:40.877 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:14:40.877 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:14:40.877 INFO:teuthology.orchestra.run.smithi119.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:14:40.877 INFO:teuthology.orchestra.run.smithi119.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:14:40.877 INFO:teuthology.orchestra.run.smithi119.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:14:40.877 INFO:teuthology.orchestra.run.smithi119.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:14:40.877 INFO:teuthology.orchestra.run.smithi119.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:14:40.878 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:14:40.878 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-29T11:14:40.909 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-03-29T11:14:40.978 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:14:41.009 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:14:41.036 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:14:41.106 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:14:41.152 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:14:41.185 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:14:41.229 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-test 2024-03-29T11:14:41.277 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:14:41.363 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:14:41.412 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:14:41.484 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:14:41.538 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:14:41.576 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:14:41.625 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:14:41.680 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:14:41.790 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:27 ago on Fri 29 Mar 2024 11:14:14 AM UTC. 2024-03-29T11:14:41.864 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:14:41.922 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:14:41.941 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-29T11:14:41.941 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-29T11:14:41.942 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-03-29T11:14:41.942 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-29T11:14:41.942 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-03-29T11:14:41.942 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:14:41.942 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-03-29T11:14:41.942 INFO:teuthology.orchestra.run.smithi119.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:14:41.942 INFO:teuthology.orchestra.run.smithi119.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:14:41.942 INFO:teuthology.orchestra.run.smithi119.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:14:41.942 INFO:teuthology.orchestra.run.smithi119.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:14:41.943 INFO:teuthology.orchestra.run.smithi119.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:14:41.943 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:14:41.943 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-03-29T11:14:41.943 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-29T11:14:41.943 INFO:teuthology.orchestra.run.smithi119.stdout:Install 6 Packages 2024-03-29T11:14:41.943 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:14:41.943 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 48 M 2024-03-29T11:14:41.944 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 204 M 2024-03-29T11:14:41.944 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-03-29T11:14:41.960 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:14:41.995 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:14:42.025 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:14:42.101 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:14:42.135 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:14:42.160 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:14:42.201 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:14:42.235 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:14:42.270 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:14:42.279 INFO:teuthology.orchestra.run.smithi119.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.1 MB/s | 217 kB 00:00 2024-03-29T11:14:42.359 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:14:42.363 INFO:teuthology.orchestra.run.smithi119.stdout:(2/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 574 kB/s | 163 kB 00:00 2024-03-29T11:14:42.388 INFO:teuthology.orchestra.run.smithi119.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 2.7 MB/s | 300 kB 00:00 2024-03-29T11:14:42.401 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:14:42.422 INFO:teuthology.orchestra.run.smithi119.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.0 MB/s | 63 kB 00:00 2024-03-29T11:14:42.473 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:14:42.614 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:14:42.737 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:14:42.822 INFO:teuthology.orchestra.run.smithi119.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 432 kB/s | 187 kB 00:00 2024-03-29T11:14:42.901 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:14:42.930 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:14:42.956 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:14:42.956 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:14:42.956 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:14:42.957 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:14:42.957 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:14:42.957 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:14:44.398 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:14:44.425 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:14:44.425 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:14:44.425 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:14:44.425 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:14:44.425 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:14:44.425 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:14:44.522 INFO:teuthology.orchestra.run.smithi119.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 19 MB/s | 47 MB 00:02 2024-03-29T11:14:44.525 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:14:44.526 INFO:teuthology.orchestra.run.smithi119.stdout:Total 19 MB/s | 48 MB 00:02 2024-03-29T11:14:44.541 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-03-29T11:14:44.562 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-03-29T11:14:44.562 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-03-29T11:14:44.865 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:14:44.888 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:14:44.888 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:14:44.888 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:14:44.888 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:14:44.888 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:14:44.888 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:14:44.944 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-03-29T11:14:44.944 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-03-29T11:14:45.091 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:14:45.114 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:14:45.114 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:14:45.114 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:14:45.114 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:14:45.114 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:14:45.114 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:14:45.179 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:14:45.531 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-03-29T11:14:45.614 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:14:45.652 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:14:45.693 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:14:45.739 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:14:45.767 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:14:46.918 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:14:46.918 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:14:46.919 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:14:46.919 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:14:46.919 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:14:46.919 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:14:46.919 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:14:46.919 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:14:46.919 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:14:46.919 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:14:46.919 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:14:46.919 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:14:46.919 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:14:46.919 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:14:46.919 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:14:46.919 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:14:46.920 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:14:46.920 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:14:46.920 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:14:46.920 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:14:46.920 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:14:46.920 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:14:46.920 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:14:46.920 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:14:46.920 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:14:46.920 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:14:46.920 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:14:46.920 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:14:46.920 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:14:46.920 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:14:46.921 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:14:46.921 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:14:46.921 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:14:46.921 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:14:46.921 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:14:46.921 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:14:47.950 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:14:47.950 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:14:47.950 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-03-29T11:14:47.950 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:14:47.950 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:14:47.950 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:14:47.950 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:14:47.950 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:14:47.951 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:14:47.951 INFO:teuthology.orchestra.run.smithi104.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:14:47.951 INFO:teuthology.orchestra.run.smithi104.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:14:47.951 INFO:teuthology.orchestra.run.smithi104.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:14:47.951 INFO:teuthology.orchestra.run.smithi104.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:14:47.951 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:14:47.951 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:14:47.951 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:14:47.951 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:14:47.951 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:14:47.951 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:14:47.951 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:14:47.951 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:14:47.951 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:14:47.952 INFO:teuthology.orchestra.run.smithi104.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:14:47.952 INFO:teuthology.orchestra.run.smithi104.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:14:47.952 INFO:teuthology.orchestra.run.smithi104.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:14:47.952 INFO:teuthology.orchestra.run.smithi104.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:14:47.952 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:14:47.952 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:14:47.952 INFO:teuthology.orchestra.run.smithi104.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:14:47.952 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:14:47.952 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:14:47.952 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:14:47.952 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:14:47.952 INFO:teuthology.orchestra.run.smithi104.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:14:47.952 INFO:teuthology.orchestra.run.smithi104.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:14:47.952 INFO:teuthology.orchestra.run.smithi104.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:14:47.952 INFO:teuthology.orchestra.run.smithi104.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:14:47.952 INFO:teuthology.orchestra.run.smithi104.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:14:47.953 INFO:teuthology.orchestra.run.smithi104.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:14:47.953 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:14:47.953 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-03-29T11:14:48.307 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-base 2024-03-29T11:14:48.872 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:00:56 ago on Fri 29 Mar 2024 11:13:52 AM UTC. 2024-03-29T11:14:48.972 INFO:teuthology.orchestra.run.smithi104.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:14:49.024 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-03-29T11:14:49.025 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-03-29T11:14:49.025 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-03-29T11:14:49.106 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install cephadm 2024-03-29T11:14:49.666 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:00:57 ago on Fri 29 Mar 2024 11:13:52 AM UTC. 2024-03-29T11:14:49.817 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-03-29T11:14:49.818 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-03-29T11:14:49.818 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-03-29T11:14:49.818 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-03-29T11:14:49.818 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-03-29T11:14:49.818 INFO:teuthology.orchestra.run.smithi104.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:14:49.818 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:14:49.818 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-03-29T11:14:49.819 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-03-29T11:14:49.819 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-03-29T11:14:49.819 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:14:49.819 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 747 k 2024-03-29T11:14:49.819 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 754 k 2024-03-29T11:14:49.819 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-03-29T11:14:50.245 INFO:teuthology.orchestra.run.smithi104.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.7 MB/s | 747 kB 00:00 2024-03-29T11:14:50.245 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:14:50.246 INFO:teuthology.orchestra.run.smithi104.stdout:Total 1.7 MB/s | 747 kB 00:00 2024-03-29T11:14:50.246 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-03-29T11:14:50.251 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-03-29T11:14:50.251 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-03-29T11:14:50.257 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-03-29T11:14:50.257 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-03-29T11:14:50.268 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:14:50.331 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-03-29T11:14:50.341 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:14:50.677 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:14:50.765 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:14:51.122 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:14:51.403 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:14:51.403 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:14:51.403 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-03-29T11:14:51.404 INFO:teuthology.orchestra.run.smithi104.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:14:51.404 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:14:51.404 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-03-29T11:14:51.638 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:14:52.212 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:00 ago on Fri 29 Mar 2024 11:13:52 AM UTC. 2024-03-29T11:14:52.366 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-03-29T11:14:52.367 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-03-29T11:14:52.367 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repo Size 2024-03-29T11:14:52.368 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-03-29T11:14:52.368 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-03-29T11:14:52.368 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:14:52.368 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:14:52.368 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-03-29T11:14:52.368 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-03-29T11:14:52.368 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-03-29T11:14:52.368 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:14:52.368 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 145 k 2024-03-29T11:14:52.368 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 436 k 2024-03-29T11:14:52.368 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-03-29T11:14:52.600 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 628 kB/s | 145 kB 00:00 2024-03-29T11:14:52.600 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:14:52.601 INFO:teuthology.orchestra.run.smithi104.stdout:Total 624 kB/s | 145 kB 00:00 2024-03-29T11:14:52.601 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-03-29T11:14:52.608 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-03-29T11:14:52.608 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-03-29T11:14:52.660 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-03-29T11:14:52.661 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-03-29T11:14:52.743 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:14:52.743 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:14:52.743 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:14:52.743 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:14:52.743 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:14:52.743 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:14:52.795 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-03-29T11:14:52.859 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:14:52.883 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:14:52.884 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:14:52.884 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:14:52.884 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:14:53.110 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:14:53.110 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:14:53.110 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-03-29T11:14:53.110 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:14:53.110 INFO:teuthology.orchestra.run.smithi119.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:14:53.110 INFO:teuthology.orchestra.run.smithi119.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:14:53.110 INFO:teuthology.orchestra.run.smithi119.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:14:53.110 INFO:teuthology.orchestra.run.smithi119.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:14:53.111 INFO:teuthology.orchestra.run.smithi119.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:14:53.111 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:14:53.111 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-29T11:14:53.423 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph 2024-03-29T11:14:53.846 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:14:53.846 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:14:53.846 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-03-29T11:14:53.846 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:14:53.846 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:14:53.846 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-03-29T11:14:54.016 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:40 ago on Fri 29 Mar 2024 11:14:14 AM UTC. 2024-03-29T11:14:54.091 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-mgr 2024-03-29T11:14:54.177 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-29T11:14:54.182 INFO:teuthology.orchestra.run.smithi119.stdout:========================================================================================= 2024-03-29T11:14:54.183 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-03-29T11:14:54.183 INFO:teuthology.orchestra.run.smithi119.stdout:========================================================================================= 2024-03-29T11:14:54.183 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-03-29T11:14:54.183 INFO:teuthology.orchestra.run.smithi119.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:14:54.183 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-03-29T11:14:54.183 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:14:54.183 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:14:54.183 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:14:54.183 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:14:54.183 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:14:54.183 INFO:teuthology.orchestra.run.smithi119.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:14:54.183 INFO:teuthology.orchestra.run.smithi119.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:14:54.183 INFO:teuthology.orchestra.run.smithi119.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:14:54.184 INFO:teuthology.orchestra.run.smithi119.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:14:54.184 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:14:54.184 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:14:54.184 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:14:54.184 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:14:54.184 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:14:54.184 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:14:54.184 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:14:54.184 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:14:54.184 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:14:54.184 INFO:teuthology.orchestra.run.smithi119.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:14:54.184 INFO:teuthology.orchestra.run.smithi119.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:14:54.184 INFO:teuthology.orchestra.run.smithi119.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:14:54.184 INFO:teuthology.orchestra.run.smithi119.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:14:54.184 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:14:54.185 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:14:54.185 INFO:teuthology.orchestra.run.smithi119.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:14:54.185 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:14:54.185 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:14:54.185 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:14:54.185 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:14:54.185 INFO:teuthology.orchestra.run.smithi119.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:14:54.185 INFO:teuthology.orchestra.run.smithi119.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:14:54.185 INFO:teuthology.orchestra.run.smithi119.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:14:54.185 INFO:teuthology.orchestra.run.smithi119.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:14:54.185 INFO:teuthology.orchestra.run.smithi119.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:14:54.185 INFO:teuthology.orchestra.run.smithi119.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:14:54.185 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:14:54.185 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-03-29T11:14:54.185 INFO:teuthology.orchestra.run.smithi119.stdout:========================================================================================= 2024-03-29T11:14:54.186 INFO:teuthology.orchestra.run.smithi119.stdout:Install 36 Packages 2024-03-29T11:14:54.186 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:14:54.186 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 30 M 2024-03-29T11:14:54.186 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 107 M 2024-03-29T11:14:54.186 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-03-29T11:14:54.664 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:02 ago on Fri 29 Mar 2024 11:13:52 AM UTC. 2024-03-29T11:14:54.766 INFO:teuthology.orchestra.run.smithi104.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:14:54.820 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-03-29T11:14:54.821 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-03-29T11:14:54.821 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-03-29T11:14:54.897 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:14:54.949 INFO:teuthology.orchestra.run.smithi119.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 29 kB/s | 6.6 kB 00:00 2024-03-29T11:14:55.092 INFO:teuthology.orchestra.run.smithi119.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 5.7 MB/s | 2.1 MB 00:00 2024-03-29T11:14:55.233 INFO:teuthology.orchestra.run.smithi119.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 2.9 MB/s | 1.5 MB 00:00 2024-03-29T11:14:55.342 INFO:teuthology.orchestra.run.smithi119.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 2.2 MB/s | 247 kB 00:00 2024-03-29T11:14:55.458 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:03 ago on Fri 29 Mar 2024 11:13:52 AM UTC. 2024-03-29T11:14:55.467 INFO:teuthology.orchestra.run.smithi119.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-29T11:14:55.509 INFO:teuthology.orchestra.run.smithi119.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-29T11:14:55.543 INFO:teuthology.orchestra.run.smithi119.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 950 kB/s | 32 kB 00:00 2024-03-29T11:14:55.576 INFO:teuthology.orchestra.run.smithi119.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-29T11:14:55.608 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-03-29T11:14:55.609 INFO:teuthology.orchestra.run.smithi104.stdout:====================================================================================== 2024-03-29T11:14:55.609 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-03-29T11:14:55.609 INFO:teuthology.orchestra.run.smithi104.stdout:====================================================================================== 2024-03-29T11:14:55.610 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-03-29T11:14:55.610 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:14:55.610 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-03-29T11:14:55.610 INFO:teuthology.orchestra.run.smithi104.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:14:55.610 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:14:55.610 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:14:55.610 INFO:teuthology.orchestra.run.smithi104.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:14:55.610 INFO:teuthology.orchestra.run.smithi104.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:14:55.610 INFO:teuthology.orchestra.run.smithi104.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:14:55.610 INFO:teuthology.orchestra.run.smithi104.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:14:55.610 INFO:teuthology.orchestra.run.smithi104.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:14:55.610 INFO:teuthology.orchestra.run.smithi104.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:14:55.611 INFO:teuthology.orchestra.run.smithi104.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:14:55.611 INFO:teuthology.orchestra.run.smithi104.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:14:55.611 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:14:55.611 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-03-29T11:14:55.611 INFO:teuthology.orchestra.run.smithi104.stdout:====================================================================================== 2024-03-29T11:14:55.611 INFO:teuthology.orchestra.run.smithi104.stdout:Install 12 Packages 2024-03-29T11:14:55.611 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:14:55.612 INFO:teuthology.orchestra.run.smithi119.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-29T11:14:55.612 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 6.9 M 2024-03-29T11:14:55.612 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 26 M 2024-03-29T11:14:55.612 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-03-29T11:14:55.652 INFO:teuthology.orchestra.run.smithi119.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-29T11:14:55.727 INFO:teuthology.orchestra.run.smithi119.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-29T11:14:55.752 INFO:teuthology.orchestra.run.smithi119.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 4.0 MB/s | 103 kB 00:00 2024-03-29T11:14:55.786 INFO:teuthology.orchestra.run.smithi119.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-29T11:14:55.820 INFO:teuthology.orchestra.run.smithi119.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-29T11:14:56.011 INFO:teuthology.orchestra.run.smithi119.stdout:(15/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 18 MB/s | 17 MB 00:00 2024-03-29T11:14:56.078 INFO:teuthology.orchestra.run.smithi119.stdout:(16/36): python3-requests-2.25.1-8.el9.noarch.r 483 kB/s | 125 kB 00:00 2024-03-29T11:14:56.104 INFO:teuthology.orchestra.run.smithi119.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 243 kB/s | 22 kB 00:00 2024-03-29T11:14:56.111 INFO:teuthology.orchestra.run.smithi104.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 85 kB/s | 15 kB 00:00 2024-03-29T11:14:56.169 INFO:teuthology.orchestra.run.smithi104.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 131 kB/s | 31 kB 00:00 2024-03-29T11:14:56.170 INFO:teuthology.orchestra.run.smithi119.stdout:(18/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 3.8 MB/s | 4.7 MB 00:01 2024-03-29T11:14:56.236 INFO:teuthology.orchestra.run.smithi104.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 3.5 MB/s | 238 kB 00:00 2024-03-29T11:14:56.254 INFO:teuthology.orchestra.run.smithi119.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 289 kB/s | 43 kB 00:00 2024-03-29T11:14:56.314 INFO:teuthology.orchestra.run.smithi104.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.7 MB/s | 1.0 MB 00:00 2024-03-29T11:14:56.314 INFO:teuthology.orchestra.run.smithi119.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 722 kB/s | 151 kB 00:00 2024-03-29T11:14:56.330 INFO:teuthology.orchestra.run.smithi119.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 257 kB/s | 11 kB 00:00 2024-03-29T11:14:56.355 INFO:teuthology.orchestra.run.smithi119.stdout:(22/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 933 kB/s | 172 kB 00:00 2024-03-29T11:14:56.380 INFO:teuthology.orchestra.run.smithi119.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 351 kB/s | 18 kB 00:00 2024-03-29T11:14:56.406 INFO:teuthology.orchestra.run.smithi119.stdout:(24/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.3 MB/s | 358 kB 00:00 2024-03-29T11:14:56.431 INFO:teuthology.orchestra.run.smithi119.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 307 kB/s | 23 kB 00:00 2024-03-29T11:14:56.447 INFO:teuthology.orchestra.run.smithi119.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 290 kB/s | 19 kB 00:00 2024-03-29T11:14:56.465 INFO:teuthology.orchestra.run.smithi119.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 335 kB/s | 20 kB 00:00 2024-03-29T11:14:56.473 INFO:teuthology.orchestra.run.smithi104.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 3.3 MB/s | 1.8 MB 00:00 2024-03-29T11:14:56.481 INFO:teuthology.orchestra.run.smithi119.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 922 kB/s | 46 kB 00:00 2024-03-29T11:14:56.507 INFO:teuthology.orchestra.run.smithi119.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-03-29T11:14:56.524 INFO:teuthology.orchestra.run.smithi119.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 388 kB/s | 16 kB 00:00 2024-03-29T11:14:56.549 INFO:teuthology.orchestra.run.smithi119.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-03-29T11:14:56.566 INFO:teuthology.orchestra.run.smithi119.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 854 kB/s | 36 kB 00:00 2024-03-29T11:14:56.608 INFO:teuthology.orchestra.run.smithi119.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.8 MB/s | 230 kB 00:00 2024-03-29T11:14:56.633 INFO:teuthology.orchestra.run.smithi119.stdout:(34/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.6 MB/s | 272 kB 00:00 2024-03-29T11:14:56.658 INFO:teuthology.orchestra.run.smithi119.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 399 kB/s | 20 kB 00:00 2024-03-29T11:14:56.700 INFO:teuthology.orchestra.run.smithi119.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.1 MB/s | 427 kB 00:00 2024-03-29T11:14:56.704 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:14:56.704 INFO:teuthology.orchestra.run.smithi119.stdout:Total 12 MB/s | 30 MB 00:02 2024-03-29T11:14:56.727 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-03-29T11:14:56.746 INFO:teuthology.orchestra.run.smithi104.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 71 kB/s | 19 kB 00:00 2024-03-29T11:14:56.768 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-03-29T11:14:56.768 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-03-29T11:14:56.871 INFO:teuthology.orchestra.run.smithi104.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 938 kB/s | 548 kB 00:00 2024-03-29T11:14:56.980 INFO:teuthology.orchestra.run.smithi104.stdout:(8/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.1 MB/s | 863 kB 00:00 2024-03-29T11:14:57.160 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-03-29T11:14:57.160 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-03-29T11:14:57.246 INFO:teuthology.orchestra.run.smithi104.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 115 kB/s | 31 kB 00:00 2024-03-29T11:14:57.330 INFO:teuthology.orchestra.run.smithi104.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 315 kB/s | 144 kB 00:00 2024-03-29T11:14:57.614 INFO:teuthology.orchestra.run.smithi104.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.3 MB/s | 2.0 MB 00:00 2024-03-29T11:14:57.688 INFO:teuthology.orchestra.run.smithi104.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 427 kB/s | 188 kB 00:00 2024-03-29T11:14:57.693 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:14:57.693 INFO:teuthology.orchestra.run.smithi104.stdout:Total 3.3 MB/s | 6.9 MB 00:02 2024-03-29T11:14:57.714 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-03-29T11:14:57.733 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-03-29T11:14:57.733 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-03-29T11:14:57.904 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-03-29T11:14:57.905 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-03-29T11:14:58.413 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-03-29T11:14:58.488 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:14:58.504 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-03-29T11:14:58.524 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:14:58.561 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:14:58.574 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:14:58.593 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:14:58.607 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:14:58.639 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:14:58.690 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:14:58.709 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:14:58.754 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:14:58.787 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:14:58.816 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:14:58.878 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:14:58.920 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:14:58.964 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:14:58.964 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:14:58.992 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:14:59.006 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:14:59.023 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:14:59.078 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:14:59.132 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:14:59.170 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:14:59.219 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:14:59.256 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:14:59.275 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:14:59.291 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:14:59.460 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:14:59.508 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:14:59.546 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:14:59.581 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:14:59.611 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:14:59.687 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:14:59.721 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:14:59.755 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:14:59.791 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:14:59.822 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:14:59.856 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:14:59.945 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:14:59.987 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:15:00.038 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:15:00.038 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:15:00.038 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:15:00.038 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:15:00.038 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:15:00.038 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:15:00.039 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:15:00.039 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:15:00.039 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:15:00.039 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:15:00.039 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:15:00.039 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:15:00.059 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:15:00.197 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:15:00.322 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:15:00.377 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:15:00.377 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:15:00.377 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-03-29T11:15:00.377 INFO:teuthology.orchestra.run.smithi104.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:15:00.377 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:15:00.377 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:15:00.377 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:15:00.377 INFO:teuthology.orchestra.run.smithi104.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:15:00.377 INFO:teuthology.orchestra.run.smithi104.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:15:00.377 INFO:teuthology.orchestra.run.smithi104.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:15:00.377 INFO:teuthology.orchestra.run.smithi104.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:15:00.377 INFO:teuthology.orchestra.run.smithi104.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:15:00.378 INFO:teuthology.orchestra.run.smithi104.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:15:00.378 INFO:teuthology.orchestra.run.smithi104.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:15:00.378 INFO:teuthology.orchestra.run.smithi104.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:15:00.378 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:15:00.378 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-03-29T11:15:00.487 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:15:00.517 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:15:00.542 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:15:00.542 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:15:00.542 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:15:00.542 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:15:00.543 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:15:00.543 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:15:00.651 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:15:01.210 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:09 ago on Fri 29 Mar 2024 11:13:52 AM UTC. 2024-03-29T11:15:01.363 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-03-29T11:15:01.363 INFO:teuthology.orchestra.run.smithi104.stdout:============================================================================================ 2024-03-29T11:15:01.364 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-03-29T11:15:01.364 INFO:teuthology.orchestra.run.smithi104.stdout:============================================================================================ 2024-03-29T11:15:01.364 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-03-29T11:15:01.364 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:15:01.364 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-03-29T11:15:01.364 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:15:01.364 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:15:01.364 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:15:01.364 INFO:teuthology.orchestra.run.smithi104.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:15:01.364 INFO:teuthology.orchestra.run.smithi104.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:15:01.364 INFO:teuthology.orchestra.run.smithi104.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:15:01.364 INFO:teuthology.orchestra.run.smithi104.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:15:01.364 INFO:teuthology.orchestra.run.smithi104.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:15:01.364 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:15:01.365 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:15:01.365 INFO:teuthology.orchestra.run.smithi104.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:15:01.365 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:15:01.365 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-03-29T11:15:01.365 INFO:teuthology.orchestra.run.smithi104.stdout:============================================================================================ 2024-03-29T11:15:01.365 INFO:teuthology.orchestra.run.smithi104.stdout:Install 12 Packages 2024-03-29T11:15:01.365 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:15:01.365 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 37 M 2024-03-29T11:15:01.366 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 209 M 2024-03-29T11:15:01.366 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-03-29T11:15:01.728 INFO:teuthology.orchestra.run.smithi104.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 152 kB/s | 32 kB 00:00 2024-03-29T11:15:01.761 INFO:teuthology.orchestra.run.smithi104.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 514 kB/s | 17 kB 00:00 2024-03-29T11:15:01.803 INFO:teuthology.orchestra.run.smithi104.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 831 kB/s | 34 kB 00:00 2024-03-29T11:15:01.878 INFO:teuthology.orchestra.run.smithi104.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 8.3 MB/s | 3.0 MB 00:00 2024-03-29T11:15:01.968 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:15:01.995 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:15:01.995 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:15:01.995 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:15:01.995 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:15:01.995 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:15:01.996 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:15:02.095 INFO:teuthology.orchestra.run.smithi104.stdout:(5/12): ceph-mgr-diskprediction-local-19.0.0-14 13 MB/s | 7.4 MB 00:00 2024-03-29T11:15:02.163 INFO:teuthology.orchestra.run.smithi104.stdout:(6/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 13 MB/s | 4.7 MB 00:00 2024-03-29T11:15:02.220 INFO:teuthology.orchestra.run.smithi104.stdout:(7/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 14 MB/s | 4.8 MB 00:00 2024-03-29T11:15:02.263 INFO:teuthology.orchestra.run.smithi104.stdout:(8/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.3 MB/s | 217 kB 00:00 2024-03-29T11:15:02.435 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:15:02.458 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:15:02.458 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:15:02.458 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:15:02.458 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:15:02.458 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:15:02.458 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:15:02.572 INFO:teuthology.orchestra.run.smithi104.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 45 MB/s | 16 MB 00:00 2024-03-29T11:15:02.660 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:15:02.683 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:15:02.683 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:15:02.684 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:15:02.684 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:15:02.684 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:15:02.684 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:15:02.689 INFO:teuthology.orchestra.run.smithi104.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 462 kB/s | 195 kB 00:00 2024-03-29T11:15:02.742 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:15:02.747 INFO:teuthology.orchestra.run.smithi104.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.4 MB/s | 245 kB 00:00 2024-03-29T11:15:02.989 INFO:teuthology.orchestra.run.smithi104.stdout:(12/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 803 kB 00:00 2024-03-29T11:15:02.991 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:15:02.991 INFO:teuthology.orchestra.run.smithi104.stdout:Total 23 MB/s | 37 MB 00:01 2024-03-29T11:15:03.032 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-03-29T11:15:03.055 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-03-29T11:15:03.056 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-03-29T11:15:03.426 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-03-29T11:15:03.426 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-03-29T11:15:04.150 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-03-29T11:15:04.211 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:15:04.243 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:15:04.336 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:15:04.388 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:15:04.528 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:15:04.528 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:15:04.528 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:15:04.528 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:15:04.528 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:15:04.528 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:15:04.528 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:15:04.528 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:15:04.528 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:15:04.528 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:15:04.528 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:15:04.529 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:15:04.529 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:15:04.529 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:15:04.529 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:15:04.529 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:15:04.529 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:15:04.529 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:15:04.529 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:15:04.529 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:15:04.529 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:15:04.529 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:15:04.529 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:15:04.529 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:15:04.529 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:15:04.529 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:15:04.531 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:15:04.531 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:15:04.531 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:15:04.531 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:15:04.532 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:15:04.532 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:15:04.532 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:15:04.532 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:15:04.532 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:15:04.532 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:15:04.565 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:15:04.595 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:15:04.679 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:15:05.330 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:15:05.432 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:15:05.512 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:15:05.513 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:15:05.513 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-03-29T11:15:05.513 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:15:05.513 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:15:05.513 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:15:05.513 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:15:05.513 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:15:05.513 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:15:05.513 INFO:teuthology.orchestra.run.smithi119.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:15:05.513 INFO:teuthology.orchestra.run.smithi119.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:15:05.513 INFO:teuthology.orchestra.run.smithi119.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:15:05.513 INFO:teuthology.orchestra.run.smithi119.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:15:05.513 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:15:05.513 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:15:05.514 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:15:05.514 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:15:05.514 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:15:05.514 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:15:05.514 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:15:05.514 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:15:05.514 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:15:05.514 INFO:teuthology.orchestra.run.smithi119.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:15:05.514 INFO:teuthology.orchestra.run.smithi119.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:15:05.514 INFO:teuthology.orchestra.run.smithi119.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:15:05.514 INFO:teuthology.orchestra.run.smithi119.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:15:05.514 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:15:05.514 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:15:05.515 INFO:teuthology.orchestra.run.smithi119.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:15:05.515 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:15:05.515 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:15:05.515 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:15:05.515 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:15:05.515 INFO:teuthology.orchestra.run.smithi119.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:15:05.515 INFO:teuthology.orchestra.run.smithi119.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:15:05.515 INFO:teuthology.orchestra.run.smithi119.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:15:05.515 INFO:teuthology.orchestra.run.smithi119.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:15:05.515 INFO:teuthology.orchestra.run.smithi119.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:15:05.515 INFO:teuthology.orchestra.run.smithi119.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:15:05.515 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:15:05.515 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-29T11:15:05.930 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-base 2024-03-29T11:15:06.497 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:52 ago on Fri 29 Mar 2024 11:14:14 AM UTC. 2024-03-29T11:15:06.595 INFO:teuthology.orchestra.run.smithi119.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:15:06.647 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-29T11:15:06.647 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-03-29T11:15:06.648 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-29T11:15:06.675 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:15:06.720 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install cephadm 2024-03-29T11:15:07.279 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:53 ago on Fri 29 Mar 2024 11:14:14 AM UTC. 2024-03-29T11:15:07.429 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-29T11:15:07.429 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-29T11:15:07.429 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-03-29T11:15:07.430 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-29T11:15:07.430 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-03-29T11:15:07.430 INFO:teuthology.orchestra.run.smithi119.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:15:07.430 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:15:07.430 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-03-29T11:15:07.430 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-29T11:15:07.430 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-03-29T11:15:07.430 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:15:07.430 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 747 k 2024-03-29T11:15:07.430 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 754 k 2024-03-29T11:15:07.430 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-03-29T11:15:07.709 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:15:07.709 INFO:teuthology.orchestra.run.smithi119.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 2.6 MB/s | 747 kB 00:00 2024-03-29T11:15:07.709 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:15:07.710 INFO:teuthology.orchestra.run.smithi119.stdout:Total 2.6 MB/s | 747 kB 00:00 2024-03-29T11:15:07.710 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-03-29T11:15:07.715 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-03-29T11:15:07.715 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-03-29T11:15:07.721 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-03-29T11:15:07.721 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-03-29T11:15:07.743 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:15:07.794 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-03-29T11:15:08.250 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:15:08.331 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:15:08.712 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:15:08.986 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:15:08.986 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:15:08.986 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-03-29T11:15:08.986 INFO:teuthology.orchestra.run.smithi119.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:15:08.986 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:15:08.986 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-29T11:15:09.207 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:15:09.774 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:55 ago on Fri 29 Mar 2024 11:14:14 AM UTC. 2024-03-29T11:15:09.929 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-29T11:15:09.929 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-29T11:15:09.929 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repo Size 2024-03-29T11:15:09.929 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-29T11:15:09.930 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-03-29T11:15:09.930 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:15:09.930 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:15:09.930 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-03-29T11:15:09.930 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-29T11:15:09.930 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-03-29T11:15:09.930 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:15:09.930 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 145 k 2024-03-29T11:15:09.930 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 436 k 2024-03-29T11:15:09.930 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-03-29T11:15:10.174 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:15:10.175 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:15:10.175 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:15:10.175 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:15:10.175 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:15:10.175 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:15:10.176 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:15:10.176 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:15:10.176 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:15:10.176 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:15:10.177 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:15:10.177 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:15:10.273 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 424 kB/s | 145 kB 00:00 2024-03-29T11:15:10.274 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:15:10.274 INFO:teuthology.orchestra.run.smithi119.stdout:Total 422 kB/s | 145 kB 00:00 2024-03-29T11:15:10.274 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-03-29T11:15:10.282 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-03-29T11:15:10.282 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-03-29T11:15:10.334 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-03-29T11:15:10.334 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-03-29T11:15:10.474 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-03-29T11:15:10.532 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:15:10.556 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:15:10.556 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:15:10.556 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:15:10.557 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:15:10.641 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:15:10.641 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:15:10.641 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-03-29T11:15:10.641 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:15:10.642 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:15:10.642 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:15:10.642 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:15:10.642 INFO:teuthology.orchestra.run.smithi104.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:15:10.643 INFO:teuthology.orchestra.run.smithi104.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:15:10.643 INFO:teuthology.orchestra.run.smithi104.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:15:10.643 INFO:teuthology.orchestra.run.smithi104.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:15:10.643 INFO:teuthology.orchestra.run.smithi104.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:15:10.643 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:15:10.643 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:15:10.643 INFO:teuthology.orchestra.run.smithi104.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:15:10.643 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:15:10.643 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-03-29T11:15:11.046 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:15:11.527 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:15:11.527 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:15:11.527 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-03-29T11:15:11.527 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:15:11.527 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:15:11.527 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-29T11:15:11.615 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:19 ago on Fri 29 Mar 2024 11:13:52 AM UTC. 2024-03-29T11:15:11.772 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-03-29T11:15:11.774 INFO:teuthology.orchestra.run.smithi104.stdout:======================================================================================== 2024-03-29T11:15:11.774 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-03-29T11:15:11.774 INFO:teuthology.orchestra.run.smithi104.stdout:======================================================================================== 2024-03-29T11:15:11.774 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-03-29T11:15:11.774 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:15:11.774 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-03-29T11:15:11.774 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:15:11.774 INFO:teuthology.orchestra.run.smithi104.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:15:11.774 INFO:teuthology.orchestra.run.smithi104.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:15:11.774 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:15:11.774 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:15:11.774 INFO:teuthology.orchestra.run.smithi104.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:15:11.775 INFO:teuthology.orchestra.run.smithi104.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:15:11.775 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:15:11.775 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:15:11.775 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:15:11.775 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:15:11.775 INFO:teuthology.orchestra.run.smithi104.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:15:11.775 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:15:11.775 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-03-29T11:15:11.775 INFO:teuthology.orchestra.run.smithi104.stdout:======================================================================================== 2024-03-29T11:15:11.775 INFO:teuthology.orchestra.run.smithi104.stdout:Install 13 Packages 2024-03-29T11:15:11.775 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:15:11.776 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 2.0 M 2024-03-29T11:15:11.776 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 26 M 2024-03-29T11:15:11.777 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-03-29T11:15:11.777 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-mgr 2024-03-29T11:15:12.250 INFO:teuthology.orchestra.run.smithi104.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 157 kB/s | 18 kB 00:00 2024-03-29T11:15:12.275 INFO:teuthology.orchestra.run.smithi104.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 174 kB/s | 24 kB 00:00 2024-03-29T11:15:12.325 INFO:teuthology.orchestra.run.smithi104.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-29T11:15:12.349 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:58 ago on Fri 29 Mar 2024 11:14:14 AM UTC. 2024-03-29T11:15:12.351 INFO:teuthology.orchestra.run.smithi104.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-29T11:15:12.376 INFO:teuthology.orchestra.run.smithi104.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T11:15:12.401 INFO:teuthology.orchestra.run.smithi104.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 859 kB/s | 43 kB 00:00 2024-03-29T11:15:12.449 INFO:teuthology.orchestra.run.smithi119.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:15:12.501 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-29T11:15:12.502 INFO:teuthology.orchestra.run.smithi104.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 142 kB/s | 52 kB 00:00 2024-03-29T11:15:12.502 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-03-29T11:15:12.502 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-29T11:15:12.544 INFO:teuthology.orchestra.run.smithi104.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 192 kB/s | 32 kB 00:00 2024-03-29T11:15:12.569 INFO:teuthology.orchestra.run.smithi104.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 89 kB/s | 15 kB 00:00 2024-03-29T11:15:12.573 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:15:12.619 INFO:teuthology.orchestra.run.smithi104.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 1.7 MB/s | 203 kB 00:00 2024-03-29T11:15:12.686 INFO:teuthology.orchestra.run.smithi104.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 7.2 MB/s | 1.0 MB 00:00 2024-03-29T11:15:12.711 INFO:teuthology.orchestra.run.smithi104.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 417 kB/s | 59 kB 00:00 2024-03-29T11:15:12.737 INFO:teuthology.orchestra.run.smithi104.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 771 kB/s | 90 kB 00:00 2024-03-29T11:15:12.740 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:15:12.741 INFO:teuthology.orchestra.run.smithi104.stdout:Total 2.1 MB/s | 2.0 MB 00:00 2024-03-29T11:15:12.741 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-03-29T11:15:12.754 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-03-29T11:15:12.754 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-03-29T11:15:12.845 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-03-29T11:15:12.845 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-03-29T11:15:13.135 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:59 ago on Fri 29 Mar 2024 11:14:14 AM UTC. 2024-03-29T11:15:13.284 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-29T11:15:13.285 INFO:teuthology.orchestra.run.smithi119.stdout:====================================================================================== 2024-03-29T11:15:13.285 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-03-29T11:15:13.285 INFO:teuthology.orchestra.run.smithi119.stdout:====================================================================================== 2024-03-29T11:15:13.285 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-03-29T11:15:13.285 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:15:13.286 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-03-29T11:15:13.286 INFO:teuthology.orchestra.run.smithi119.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:15:13.286 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:15:13.286 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:15:13.286 INFO:teuthology.orchestra.run.smithi119.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:15:13.286 INFO:teuthology.orchestra.run.smithi119.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:15:13.286 INFO:teuthology.orchestra.run.smithi119.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:15:13.286 INFO:teuthology.orchestra.run.smithi119.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:15:13.286 INFO:teuthology.orchestra.run.smithi119.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:15:13.286 INFO:teuthology.orchestra.run.smithi119.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:15:13.286 INFO:teuthology.orchestra.run.smithi119.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:15:13.286 INFO:teuthology.orchestra.run.smithi119.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:15:13.286 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:15:13.286 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-03-29T11:15:13.287 INFO:teuthology.orchestra.run.smithi119.stdout:====================================================================================== 2024-03-29T11:15:13.287 INFO:teuthology.orchestra.run.smithi119.stdout:Install 12 Packages 2024-03-29T11:15:13.287 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:15:13.287 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 6.9 M 2024-03-29T11:15:13.287 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 26 M 2024-03-29T11:15:13.287 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-03-29T11:15:13.356 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-03-29T11:15:13.516 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:15:13.570 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:15:13.626 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:15:13.671 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:15:13.713 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:15:13.806 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:15:13.906 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:15:13.964 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:15:14.085 INFO:teuthology.orchestra.run.smithi119.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 98 kB/s | 31 kB 00:00 2024-03-29T11:15:14.111 INFO:teuthology.orchestra.run.smithi119.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 44 kB/s | 15 kB 00:00 2024-03-29T11:15:14.253 INFO:teuthology.orchestra.run.smithi119.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.6 MB/s | 238 kB 00:00 2024-03-29T11:15:14.286 INFO:teuthology.orchestra.run.smithi119.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.0 MB/s | 1.0 MB 00:00 2024-03-29T11:15:14.378 INFO:teuthology.orchestra.run.smithi119.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 2.9 MB/s | 1.8 MB 00:00 2024-03-29T11:15:14.495 INFO:teuthology.orchestra.run.smithi119.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.5 MB/s | 863 kB 00:00 2024-03-29T11:15:14.520 INFO:teuthology.orchestra.run.smithi119.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.3 MB/s | 548 kB 00:00 2024-03-29T11:15:14.546 INFO:teuthology.orchestra.run.smithi119.stdout:(8/12): grpc-data-1.46.7-10.el9.noarch.rpm 117 kB/s | 19 kB 00:00 2024-03-29T11:15:14.604 INFO:teuthology.orchestra.run.smithi119.stdout:(9/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 19 MB/s | 2.0 MB 00:00 2024-03-29T11:15:14.630 INFO:teuthology.orchestra.run.smithi119.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 1.3 MB/s | 144 kB 00:00 2024-03-29T11:15:14.647 INFO:teuthology.orchestra.run.smithi119.stdout:(11/12): python3-repoze-lru-0.7-16.el9.noarch.r 309 kB/s | 31 kB 00:00 2024-03-29T11:15:14.671 INFO:teuthology.orchestra.run.smithi119.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-03-29T11:15:14.675 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:15:14.675 INFO:teuthology.orchestra.run.smithi119.stdout:Total 4.9 MB/s | 6.9 MB 00:01 2024-03-29T11:15:14.682 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:15:14.694 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-03-29T11:15:14.710 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-03-29T11:15:14.710 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-03-29T11:15:14.736 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:15:14.770 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:15:14.818 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:15:14.839 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:15:14.871 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-03-29T11:15:14.872 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-03-29T11:15:15.398 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-03-29T11:15:15.489 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:15:15.532 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:15:15.579 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:15:15.588 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:15:15.589 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:15:15.589 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:15:15.589 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:15:15.589 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:15:15.589 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:15:15.589 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:15:15.589 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:15:15.589 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:15:15.589 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:15:15.589 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:15:15.589 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:15:15.589 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:15:15.609 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:15:15.706 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:15:15.836 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:15:15.921 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:15:15.942 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:15:15.942 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:15:15.942 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-03-29T11:15:15.942 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:15:15.942 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:15:15.942 INFO:teuthology.orchestra.run.smithi104.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:15:15.943 INFO:teuthology.orchestra.run.smithi104.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:15:15.943 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:15:15.943 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:15:15.943 INFO:teuthology.orchestra.run.smithi104.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:15:15.943 INFO:teuthology.orchestra.run.smithi104.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:15:15.943 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:15:15.943 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:15:15.943 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:15:15.943 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:15:15.943 INFO:teuthology.orchestra.run.smithi104.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:15:15.943 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:15:15.943 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-03-29T11:15:15.967 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:15:16.009 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:15:16.041 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:15:16.228 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:15:16.285 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:15:16.316 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:15:16.798 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:24 ago on Fri 29 Mar 2024 11:13:52 AM UTC. 2024-03-29T11:15:16.949 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-03-29T11:15:16.949 INFO:teuthology.orchestra.run.smithi104.stdout:======================================================================================== 2024-03-29T11:15:16.949 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-03-29T11:15:16.949 INFO:teuthology.orchestra.run.smithi104.stdout:======================================================================================== 2024-03-29T11:15:16.950 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-03-29T11:15:16.950 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:15:16.950 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-03-29T11:15:16.950 INFO:teuthology.orchestra.run.smithi104.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:15:16.950 INFO:teuthology.orchestra.run.smithi104.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:15:16.950 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:15:16.950 INFO:teuthology.orchestra.run.smithi104.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:15:16.950 INFO:teuthology.orchestra.run.smithi104.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:15:16.950 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:15:16.950 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-03-29T11:15:16.950 INFO:teuthology.orchestra.run.smithi104.stdout:======================================================================================== 2024-03-29T11:15:16.950 INFO:teuthology.orchestra.run.smithi104.stdout:Install 6 Packages 2024-03-29T11:15:16.950 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:15:16.951 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 6.8 M 2024-03-29T11:15:16.951 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 33 M 2024-03-29T11:15:16.951 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-03-29T11:15:17.118 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:15:17.119 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:15:17.119 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:15:17.119 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:15:17.119 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:15:17.119 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:15:17.119 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:15:17.119 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:15:17.119 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:15:17.119 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:15:17.119 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:15:17.119 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:15:17.467 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:15:17.467 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:15:17.467 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-03-29T11:15:17.467 INFO:teuthology.orchestra.run.smithi119.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:15:17.467 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:15:17.467 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:15:17.467 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:15:17.468 INFO:teuthology.orchestra.run.smithi119.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:15:17.468 INFO:teuthology.orchestra.run.smithi119.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:15:17.468 INFO:teuthology.orchestra.run.smithi119.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:15:17.468 INFO:teuthology.orchestra.run.smithi119.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:15:17.468 INFO:teuthology.orchestra.run.smithi119.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:15:17.468 INFO:teuthology.orchestra.run.smithi119.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:15:17.468 INFO:teuthology.orchestra.run.smithi119.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:15:17.468 INFO:teuthology.orchestra.run.smithi119.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:15:17.468 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:15:17.468 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-29T11:15:17.470 INFO:teuthology.orchestra.run.smithi104.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 2.1 MB/s | 248 kB 00:00 2024-03-29T11:15:17.645 INFO:teuthology.orchestra.run.smithi104.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2024-03-29T11:15:17.722 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:15:17.762 INFO:teuthology.orchestra.run.smithi104.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 337 kB/s | 138 kB 00:00 2024-03-29T11:15:17.937 INFO:teuthology.orchestra.run.smithi104.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 198 kB/s | 58 kB 00:00 2024-03-29T11:15:17.987 INFO:teuthology.orchestra.run.smithi104.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.0 MB/s | 548 kB 00:00 2024-03-29T11:15:18.046 INFO:teuthology.orchestra.run.smithi104.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 180 kB/s | 51 kB 00:00 2024-03-29T11:15:18.049 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:15:18.049 INFO:teuthology.orchestra.run.smithi104.stdout:Total 6.2 MB/s | 6.8 MB 00:01 2024-03-29T11:15:18.062 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-03-29T11:15:18.071 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-03-29T11:15:18.071 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-03-29T11:15:18.159 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-03-29T11:15:18.160 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-03-29T11:15:18.278 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:04 ago on Fri 29 Mar 2024 11:14:14 AM UTC. 2024-03-29T11:15:18.428 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-29T11:15:18.429 INFO:teuthology.orchestra.run.smithi119.stdout:============================================================================================ 2024-03-29T11:15:18.429 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-03-29T11:15:18.429 INFO:teuthology.orchestra.run.smithi119.stdout:============================================================================================ 2024-03-29T11:15:18.430 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-03-29T11:15:18.430 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:15:18.430 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-03-29T11:15:18.430 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:15:18.430 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:15:18.430 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:15:18.430 INFO:teuthology.orchestra.run.smithi119.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:15:18.430 INFO:teuthology.orchestra.run.smithi119.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:15:18.430 INFO:teuthology.orchestra.run.smithi119.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:15:18.430 INFO:teuthology.orchestra.run.smithi119.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:15:18.430 INFO:teuthology.orchestra.run.smithi119.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:15:18.430 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:15:18.430 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:15:18.431 INFO:teuthology.orchestra.run.smithi119.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:15:18.431 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:15:18.431 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-03-29T11:15:18.431 INFO:teuthology.orchestra.run.smithi119.stdout:============================================================================================ 2024-03-29T11:15:18.431 INFO:teuthology.orchestra.run.smithi119.stdout:Install 12 Packages 2024-03-29T11:15:18.431 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:15:18.431 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 37 M 2024-03-29T11:15:18.432 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 209 M 2024-03-29T11:15:18.432 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-03-29T11:15:18.465 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-03-29T11:15:18.634 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:15:18.669 INFO:teuthology.orchestra.run.smithi119.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 237 kB/s | 32 kB 00:00 2024-03-29T11:15:18.689 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:15:18.695 INFO:teuthology.orchestra.run.smithi119.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 722 kB/s | 17 kB 00:00 2024-03-29T11:15:18.720 INFO:teuthology.orchestra.run.smithi119.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.4 MB/s | 34 kB 00:00 2024-03-29T11:15:18.770 INFO:teuthology.orchestra.run.smithi119.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 13 MB/s | 3.0 MB 00:00 2024-03-29T11:15:18.962 INFO:teuthology.orchestra.run.smithi119.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2024-03-29T11:15:19.029 INFO:teuthology.orchestra.run.smithi119.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 4.8 MB 00:00 2024-03-29T11:15:19.054 INFO:teuthology.orchestra.run.smithi119.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-29T11:15:19.206 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:15:19.221 INFO:teuthology.orchestra.run.smithi119.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.8 MB/s | 803 kB 00:00 2024-03-29T11:15:19.291 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:15:19.352 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:15:19.376 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:15:19.388 INFO:teuthology.orchestra.run.smithi119.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 44 MB/s | 16 MB 00:00 2024-03-29T11:15:19.405 INFO:teuthology.orchestra.run.smithi119.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2024-03-29T11:15:19.597 INFO:teuthology.orchestra.run.smithi119.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.2 MB/s | 245 kB 00:00 2024-03-29T11:15:20.231 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:15:20.231 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:15:20.231 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:15:20.231 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:15:20.231 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:15:20.231 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:15:20.512 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:15:20.512 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:15:20.512 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-03-29T11:15:20.512 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:15:20.513 INFO:teuthology.orchestra.run.smithi104.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:15:20.513 INFO:teuthology.orchestra.run.smithi104.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:15:20.513 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:15:20.513 INFO:teuthology.orchestra.run.smithi104.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:15:20.513 INFO:teuthology.orchestra.run.smithi104.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:15:20.513 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:15:20.513 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-03-29T11:15:20.808 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-fuse 2024-03-29T11:15:20.939 INFO:teuthology.orchestra.run.smithi119.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 3.1 MB/s | 7.4 MB 00:02 2024-03-29T11:15:20.942 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:15:20.942 INFO:teuthology.orchestra.run.smithi119.stdout:Total 15 MB/s | 37 MB 00:02 2024-03-29T11:15:20.985 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-03-29T11:15:21.009 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-03-29T11:15:21.009 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-03-29T11:15:21.366 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-03-29T11:15:21.367 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-03-29T11:15:21.369 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:29 ago on Fri 29 Mar 2024 11:13:52 AM UTC. 2024-03-29T11:15:21.520 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-03-29T11:15:21.521 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-03-29T11:15:21.521 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-03-29T11:15:21.521 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-03-29T11:15:21.521 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-03-29T11:15:21.521 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:15:21.521 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-03-29T11:15:21.521 INFO:teuthology.orchestra.run.smithi104.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:15:21.521 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:15:21.521 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-03-29T11:15:21.521 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-03-29T11:15:21.521 INFO:teuthology.orchestra.run.smithi104.stdout:Install 2 Packages 2024-03-29T11:15:21.522 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:15:21.522 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 964 k 2024-03-29T11:15:21.522 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 2.8 M 2024-03-29T11:15:21.522 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-03-29T11:15:21.647 INFO:teuthology.orchestra.run.smithi104.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 632 kB/s | 79 kB 00:00 2024-03-29T11:15:22.081 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-03-29T11:15:22.153 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:15:22.185 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:15:22.188 INFO:teuthology.orchestra.run.smithi104.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 1.3 MB/s | 885 kB 00:00 2024-03-29T11:15:22.189 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:15:22.190 INFO:teuthology.orchestra.run.smithi104.stdout:Total 1.4 MB/s | 964 kB 00:00 2024-03-29T11:15:22.190 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-03-29T11:15:22.211 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-03-29T11:15:22.211 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-03-29T11:15:22.276 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:15:22.287 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-03-29T11:15:22.288 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-03-29T11:15:22.313 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:15:22.473 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-03-29T11:15:22.490 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:15:22.520 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:15:22.596 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:15:22.622 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:15:22.696 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:15:23.273 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:15:23.359 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:15:23.359 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:15:23.381 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:15:23.671 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:15:23.671 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:15:23.672 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-03-29T11:15:23.672 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:15:23.672 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:15:23.672 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-03-29T11:15:23.966 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-volume 2024-03-29T11:15:24.539 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:32 ago on Fri 29 Mar 2024 11:13:52 AM UTC. 2024-03-29T11:15:24.620 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:15:24.693 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-03-29T11:15:24.693 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-03-29T11:15:24.693 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-03-29T11:15:24.693 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-03-29T11:15:24.693 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-03-29T11:15:24.693 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:15:24.694 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-03-29T11:15:24.694 INFO:teuthology.orchestra.run.smithi104.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:15:24.694 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:15:24.694 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-03-29T11:15:24.694 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-03-29T11:15:24.694 INFO:teuthology.orchestra.run.smithi104.stdout:Install 2 Packages 2024-03-29T11:15:24.694 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:15:24.695 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 332 k 2024-03-29T11:15:24.695 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 1.5 M 2024-03-29T11:15:24.695 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-03-29T11:15:24.817 INFO:teuthology.orchestra.run.smithi104.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 564 kB/s | 69 kB 00:00 2024-03-29T11:15:25.092 INFO:teuthology.orchestra.run.smithi104.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 664 kB/s | 264 kB 00:00 2024-03-29T11:15:25.093 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:15:25.093 INFO:teuthology.orchestra.run.smithi104.stdout:Total 832 kB/s | 332 kB 00:00 2024-03-29T11:15:25.094 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-03-29T11:15:25.102 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-03-29T11:15:25.102 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-03-29T11:15:25.143 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-03-29T11:15:25.143 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-03-29T11:15:25.290 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-03-29T11:15:25.489 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:15:25.528 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:15:25.543 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:15:25.543 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:15:25.543 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:15:25.544 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:15:25.632 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:15:25.661 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:15:26.258 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:15:26.571 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:15:26.571 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:15:26.571 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-03-29T11:15:26.571 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:15:26.571 INFO:teuthology.orchestra.run.smithi104.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:15:26.572 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:15:26.572 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-03-29T11:15:26.927 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install librados-devel 2024-03-29T11:15:27.499 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:35 ago on Fri 29 Mar 2024 11:13:52 AM UTC. 2024-03-29T11:15:27.649 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-03-29T11:15:27.650 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-03-29T11:15:27.650 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repo Size 2024-03-29T11:15:27.650 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-03-29T11:15:27.650 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-03-29T11:15:27.650 INFO:teuthology.orchestra.run.smithi104.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:15:27.650 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:15:27.650 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-03-29T11:15:27.651 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-03-29T11:15:27.651 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-03-29T11:15:27.651 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:15:27.651 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 127 k 2024-03-29T11:15:27.651 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 456 k 2024-03-29T11:15:27.651 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-03-29T11:15:27.880 INFO:teuthology.orchestra.run.smithi104.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 553 kB/s | 127 kB 00:00 2024-03-29T11:15:27.881 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:15:27.881 INFO:teuthology.orchestra.run.smithi104.stdout:Total 549 kB/s | 127 kB 00:00 2024-03-29T11:15:27.881 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-03-29T11:15:27.886 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-03-29T11:15:27.886 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-03-29T11:15:27.943 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-03-29T11:15:27.944 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-03-29T11:15:28.085 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-03-29T11:15:28.179 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:15:28.212 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:15:28.212 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:15:28.212 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:15:28.212 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:15:28.212 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:15:28.212 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:15:28.213 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:15:28.213 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:15:28.213 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:15:28.213 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:15:28.213 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:15:28.213 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:15:28.616 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:15:28.621 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:15:28.621 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:15:28.621 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-03-29T11:15:28.621 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:15:28.621 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:15:28.621 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:15:28.621 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:15:28.622 INFO:teuthology.orchestra.run.smithi119.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:15:28.622 INFO:teuthology.orchestra.run.smithi119.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:15:28.622 INFO:teuthology.orchestra.run.smithi119.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:15:28.622 INFO:teuthology.orchestra.run.smithi119.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:15:28.622 INFO:teuthology.orchestra.run.smithi119.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:15:28.622 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:15:28.622 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:15:28.622 INFO:teuthology.orchestra.run.smithi119.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:15:28.622 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:15:28.622 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-29T11:15:28.910 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:15:28.911 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:15:28.911 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-03-29T11:15:28.911 INFO:teuthology.orchestra.run.smithi104.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:15:28.911 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:15:28.911 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-03-29T11:15:29.052 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:15:29.163 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install libcephfs2 2024-03-29T11:15:29.625 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:15 ago on Fri 29 Mar 2024 11:14:14 AM UTC. 2024-03-29T11:15:29.732 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:37 ago on Fri 29 Mar 2024 11:13:52 AM UTC. 2024-03-29T11:15:29.776 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-29T11:15:29.777 INFO:teuthology.orchestra.run.smithi119.stdout:======================================================================================== 2024-03-29T11:15:29.778 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-03-29T11:15:29.778 INFO:teuthology.orchestra.run.smithi119.stdout:======================================================================================== 2024-03-29T11:15:29.778 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-03-29T11:15:29.778 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:15:29.778 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-03-29T11:15:29.778 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:15:29.778 INFO:teuthology.orchestra.run.smithi119.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:15:29.778 INFO:teuthology.orchestra.run.smithi119.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:15:29.778 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:15:29.778 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:15:29.778 INFO:teuthology.orchestra.run.smithi119.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:15:29.778 INFO:teuthology.orchestra.run.smithi119.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:15:29.779 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:15:29.779 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:15:29.779 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:15:29.779 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:15:29.779 INFO:teuthology.orchestra.run.smithi119.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:15:29.779 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:15:29.779 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-03-29T11:15:29.779 INFO:teuthology.orchestra.run.smithi119.stdout:======================================================================================== 2024-03-29T11:15:29.779 INFO:teuthology.orchestra.run.smithi119.stdout:Install 13 Packages 2024-03-29T11:15:29.779 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:15:29.780 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 2.0 M 2024-03-29T11:15:29.780 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 26 M 2024-03-29T11:15:29.780 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-03-29T11:15:29.830 INFO:teuthology.orchestra.run.smithi104.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:15:29.886 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-03-29T11:15:29.890 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-03-29T11:15:29.890 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-03-29T11:15:29.953 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install libcephfs-devel 2024-03-29T11:15:30.062 INFO:teuthology.orchestra.run.smithi119.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 196 kB/s | 24 kB 00:00 2024-03-29T11:15:30.078 INFO:teuthology.orchestra.run.smithi119.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 128 kB/s | 18 kB 00:00 2024-03-29T11:15:30.137 INFO:teuthology.orchestra.run.smithi119.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 3.0 MB/s | 174 kB 00:00 2024-03-29T11:15:30.162 INFO:teuthology.orchestra.run.smithi119.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.6 MB/s | 132 kB 00:00 2024-03-29T11:15:30.187 INFO:teuthology.orchestra.run.smithi119.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T11:15:30.213 INFO:teuthology.orchestra.run.smithi119.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 853 kB/s | 43 kB 00:00 2024-03-29T11:15:30.238 INFO:teuthology.orchestra.run.smithi119.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 174 kB/s | 52 kB 00:00 2024-03-29T11:15:30.363 INFO:teuthology.orchestra.run.smithi119.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 184 kB/s | 32 kB 00:00 2024-03-29T11:15:30.414 INFO:teuthology.orchestra.run.smithi119.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 75 kB/s | 15 kB 00:00 2024-03-29T11:15:30.521 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:38 ago on Fri 29 Mar 2024 11:13:52 AM UTC. 2024-03-29T11:15:30.589 INFO:teuthology.orchestra.run.smithi119.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 668 kB/s | 203 kB 00:00 2024-03-29T11:15:30.672 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-03-29T11:15:30.673 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-03-29T11:15:30.673 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repo Size 2024-03-29T11:15:30.673 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-03-29T11:15:30.673 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-03-29T11:15:30.673 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:15:30.673 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:15:30.673 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-03-29T11:15:30.673 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-03-29T11:15:30.673 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-03-29T11:15:30.674 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:15:30.674 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 31 k 2024-03-29T11:15:30.674 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 140 k 2024-03-29T11:15:30.674 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-03-29T11:15:30.781 INFO:teuthology.orchestra.run.smithi119.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 2.5 MB/s | 1.0 MB 00:00 2024-03-29T11:15:30.806 INFO:teuthology.orchestra.run.smithi119.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 151 kB/s | 59 kB 00:00 2024-03-29T11:15:30.831 INFO:teuthology.orchestra.run.smithi119.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 371 kB/s | 90 kB 00:00 2024-03-29T11:15:30.835 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:15:30.835 INFO:teuthology.orchestra.run.smithi119.stdout:Total 1.9 MB/s | 2.0 MB 00:01 2024-03-29T11:15:30.835 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-03-29T11:15:30.840 INFO:teuthology.orchestra.run.smithi104.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 187 kB/s | 31 kB 00:00 2024-03-29T11:15:30.841 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:15:30.841 INFO:teuthology.orchestra.run.smithi104.stdout:Total 186 kB/s | 31 kB 00:00 2024-03-29T11:15:30.841 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-03-29T11:15:30.845 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-03-29T11:15:30.845 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-03-29T11:15:30.848 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-03-29T11:15:30.848 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-03-29T11:15:30.870 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-03-29T11:15:30.870 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-03-29T11:15:30.937 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-03-29T11:15:30.939 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-03-29T11:15:30.995 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-03-29T11:15:31.099 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:15:31.446 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-03-29T11:15:31.506 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:15:31.604 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:15:31.657 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:15:31.713 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:15:31.750 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:15:31.777 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:15:31.777 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:15:31.777 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-03-29T11:15:31.777 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:15:31.778 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:15:31.778 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-03-29T11:15:31.791 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:15:31.885 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:15:31.947 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install librados2 2024-03-29T11:15:31.995 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:15:32.043 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:15:32.534 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:40 ago on Fri 29 Mar 2024 11:13:52 AM UTC. 2024-03-29T11:15:32.639 INFO:teuthology.orchestra.run.smithi104.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:15:32.691 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-03-29T11:15:32.692 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-03-29T11:15:32.692 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-03-29T11:15:32.740 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install librbd1 2024-03-29T11:15:32.764 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:15:32.825 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:15:32.858 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:15:32.906 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:15:32.928 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:15:33.323 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 11:13:52 AM UTC. 2024-03-29T11:15:33.423 INFO:teuthology.orchestra.run.smithi104.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:15:33.474 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-03-29T11:15:33.475 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-03-29T11:15:33.476 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-03-29T11:15:33.532 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install python3-rados 2024-03-29T11:15:33.676 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:15:33.677 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:15:33.677 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:15:33.677 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:15:33.677 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:15:33.677 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:15:33.677 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:15:33.677 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:15:33.677 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:15:33.677 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:15:33.677 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:15:33.677 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:15:33.678 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:15:34.065 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:15:34.065 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:15:34.065 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-03-29T11:15:34.065 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:15:34.065 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:15:34.065 INFO:teuthology.orchestra.run.smithi119.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:15:34.065 INFO:teuthology.orchestra.run.smithi119.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:15:34.065 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:15:34.066 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:15:34.066 INFO:teuthology.orchestra.run.smithi119.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:15:34.066 INFO:teuthology.orchestra.run.smithi119.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:15:34.066 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:15:34.066 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:15:34.066 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:15:34.066 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:15:34.066 INFO:teuthology.orchestra.run.smithi119.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:15:34.066 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:15:34.066 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-29T11:15:34.088 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 11:13:52 AM UTC. 2024-03-29T11:15:34.186 INFO:teuthology.orchestra.run.smithi104.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:15:34.237 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-03-29T11:15:34.238 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-03-29T11:15:34.238 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-03-29T11:15:34.295 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install python3-rgw 2024-03-29T11:15:34.325 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:15:34.855 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 11:13:52 AM UTC. 2024-03-29T11:15:34.888 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:20 ago on Fri 29 Mar 2024 11:14:14 AM UTC. 2024-03-29T11:15:34.953 INFO:teuthology.orchestra.run.smithi104.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:15:35.004 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-03-29T11:15:35.005 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-03-29T11:15:35.005 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-03-29T11:15:35.039 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-29T11:15:35.040 INFO:teuthology.orchestra.run.smithi119.stdout:======================================================================================== 2024-03-29T11:15:35.040 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-03-29T11:15:35.040 INFO:teuthology.orchestra.run.smithi119.stdout:======================================================================================== 2024-03-29T11:15:35.040 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-03-29T11:15:35.040 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:15:35.040 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-03-29T11:15:35.040 INFO:teuthology.orchestra.run.smithi119.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:15:35.040 INFO:teuthology.orchestra.run.smithi119.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:15:35.040 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:15:35.040 INFO:teuthology.orchestra.run.smithi119.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:15:35.041 INFO:teuthology.orchestra.run.smithi119.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:15:35.041 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:15:35.041 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-03-29T11:15:35.041 INFO:teuthology.orchestra.run.smithi119.stdout:======================================================================================== 2024-03-29T11:15:35.041 INFO:teuthology.orchestra.run.smithi119.stdout:Install 6 Packages 2024-03-29T11:15:35.041 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:15:35.041 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 6.8 M 2024-03-29T11:15:35.042 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 33 M 2024-03-29T11:15:35.042 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-03-29T11:15:35.053 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install python3-cephfs 2024-03-29T11:15:35.576 INFO:teuthology.orchestra.run.smithi119.stdout:(1/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 578 kB/s | 138 kB 00:00 2024-03-29T11:15:35.614 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 11:13:52 AM UTC. 2024-03-29T11:15:35.668 INFO:teuthology.orchestra.run.smithi119.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 5.8 MB 00:00 2024-03-29T11:15:35.693 INFO:teuthology.orchestra.run.smithi119.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 697 kB/s | 248 kB 00:00 2024-03-29T11:15:35.712 INFO:teuthology.orchestra.run.smithi104.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:15:35.764 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-03-29T11:15:35.765 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-03-29T11:15:35.765 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-03-29T11:15:35.821 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install python3-rbd 2024-03-29T11:15:36.377 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Mar 2024 11:13:52 AM UTC. 2024-03-29T11:15:36.474 INFO:teuthology.orchestra.run.smithi104.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:15:36.527 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-03-29T11:15:36.528 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-03-29T11:15:36.528 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-03-29T11:15:36.578 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install rbd-fuse 2024-03-29T11:15:37.140 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:45 ago on Fri 29 Mar 2024 11:13:52 AM UTC. 2024-03-29T11:15:37.288 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-03-29T11:15:37.288 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-03-29T11:15:37.288 INFO:teuthology.orchestra.run.smithi104.stdout: Package Architecture Version Repository Size 2024-03-29T11:15:37.289 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-03-29T11:15:37.289 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-03-29T11:15:37.289 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:15:37.289 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:15:37.289 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-03-29T11:15:37.289 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-03-29T11:15:37.289 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-03-29T11:15:37.289 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:15:37.289 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 85 k 2024-03-29T11:15:37.289 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 231 k 2024-03-29T11:15:37.289 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-03-29T11:15:37.495 INFO:teuthology.orchestra.run.smithi104.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 414 kB/s | 85 kB 00:00 2024-03-29T11:15:37.496 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:15:37.496 INFO:teuthology.orchestra.run.smithi104.stdout:Total 412 kB/s | 85 kB 00:00 2024-03-29T11:15:37.496 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-03-29T11:15:37.503 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-03-29T11:15:37.503 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-03-29T11:15:37.559 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-03-29T11:15:37.559 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-03-29T11:15:37.718 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-03-29T11:15:37.819 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:15:38.246 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:15:38.527 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:15:38.527 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:15:38.527 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-03-29T11:15:38.528 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:15:38.528 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:15:38.528 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-03-29T11:15:38.777 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install rbd-mirror 2024-03-29T11:15:39.342 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:47 ago on Fri 29 Mar 2024 11:13:52 AM UTC. 2024-03-29T11:15:39.495 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-03-29T11:15:39.496 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-03-29T11:15:39.496 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repo Size 2024-03-29T11:15:39.496 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-03-29T11:15:39.497 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-03-29T11:15:39.497 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:15:39.497 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:15:39.497 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-03-29T11:15:39.497 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-03-29T11:15:39.497 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-03-29T11:15:39.497 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:15:39.497 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 3.1 M 2024-03-29T11:15:39.497 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 13 M 2024-03-29T11:15:39.497 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-03-29T11:15:40.981 INFO:teuthology.orchestra.run.smithi104.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 2.1 MB/s | 3.1 MB 00:01 2024-03-29T11:15:40.981 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:15:40.981 INFO:teuthology.orchestra.run.smithi104.stdout:Total 2.1 MB/s | 3.1 MB 00:01 2024-03-29T11:15:40.982 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-03-29T11:15:40.992 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-03-29T11:15:40.992 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-03-29T11:15:41.067 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-03-29T11:15:41.067 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-03-29T11:15:41.500 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-03-29T11:15:41.564 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:15:41.589 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:15:41.589 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:15:41.589 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:15:41.590 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:15:41.590 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:15:41.590 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:15:42.660 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:15:42.660 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:15:42.660 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-03-29T11:15:42.660 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:15:42.660 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:15:42.660 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-03-29T11:15:42.900 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install rbd-nbd 2024-03-29T11:15:43.471 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:51 ago on Fri 29 Mar 2024 11:13:52 AM UTC. 2024-03-29T11:15:43.622 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-03-29T11:15:43.623 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-03-29T11:15:43.623 INFO:teuthology.orchestra.run.smithi104.stdout: Package Architecture Version Repository Size 2024-03-29T11:15:43.623 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-03-29T11:15:43.623 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-03-29T11:15:43.623 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:15:43.623 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:15:43.624 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-03-29T11:15:43.624 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-03-29T11:15:43.624 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-03-29T11:15:43.624 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:15:43.624 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 171 k 2024-03-29T11:15:43.624 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 494 k 2024-03-29T11:15:43.624 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-03-29T11:15:43.832 INFO:teuthology.orchestra.run.smithi104.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 828 kB/s | 171 kB 00:00 2024-03-29T11:15:43.832 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:15:43.832 INFO:teuthology.orchestra.run.smithi104.stdout:Total 821 kB/s | 171 kB 00:00 2024-03-29T11:15:43.833 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-03-29T11:15:43.840 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-03-29T11:15:43.841 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-03-29T11:15:43.896 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-03-29T11:15:43.896 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-03-29T11:15:44.048 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-03-29T11:15:44.149 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:15:44.602 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:15:44.874 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:15:44.874 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:15:44.875 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-03-29T11:15:44.875 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:15:44.875 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:15:44.875 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-03-29T11:15:45.103 DEBUG:teuthology.parallel:result is None 2024-03-29T11:16:07.672 INFO:teuthology.orchestra.run.smithi119.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 17 kB/s | 548 kB 00:32 2024-03-29T11:16:40.359 INFO:teuthology.orchestra.run.smithi119.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 807 B/s | 51 kB 01:04 2024-03-29T11:17:44.906 INFO:teuthology.orchestra.run.smithi119.stdout:[MIRROR] python3-natsort-7.1.1-5.el9.noarch.rpm: Curl error (28): Timeout was reached for http://epel.stl.us.ssimn.org/9/Everything/x86_64/Packages/p/python3-natsort-7.1.1-5.el9.noarch.rpm [Failed to connect to epel.stl.us.ssimn.org port 80: Connection timed out] 2024-03-29T11:18:17.002 INFO:teuthology.orchestra.run.smithi119.stdout:(6/6): python3-natsort-7.1.1-5.el9.noarch.rpm 366 B/s | 58 kB 02:41 2024-03-29T11:18:17.003 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:17.003 INFO:teuthology.orchestra.run.smithi119.stdout:Total 43 kB/s | 6.8 MB 02:41 2024-03-29T11:18:17.015 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-03-29T11:18:17.027 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-03-29T11:18:17.027 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-03-29T11:18:17.118 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-03-29T11:18:17.120 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-03-29T11:18:17.413 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-03-29T11:18:17.581 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:18:17.636 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:18:18.148 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:18:18.238 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:18:18.300 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:18:18.324 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:18:19.075 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:18:19.075 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:18:19.075 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:18:19.075 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:18:19.075 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:18:19.075 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:18:19.394 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:18:19.395 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:19.395 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-03-29T11:18:19.395 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:19.395 INFO:teuthology.orchestra.run.smithi119.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:18:19.395 INFO:teuthology.orchestra.run.smithi119.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:18:19.395 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:18:19.395 INFO:teuthology.orchestra.run.smithi119.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:18:19.395 INFO:teuthology.orchestra.run.smithi119.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:18:19.395 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:19.395 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-29T11:18:19.633 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-fuse 2024-03-29T11:18:20.195 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:04:06 ago on Fri 29 Mar 2024 11:14:14 AM UTC. 2024-03-29T11:18:20.345 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-29T11:18:20.346 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-29T11:18:20.346 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-03-29T11:18:20.346 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-29T11:18:20.346 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-03-29T11:18:20.346 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:18:20.346 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-03-29T11:18:20.347 INFO:teuthology.orchestra.run.smithi119.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:18:20.347 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:20.347 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-03-29T11:18:20.347 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-29T11:18:20.347 INFO:teuthology.orchestra.run.smithi119.stdout:Install 2 Packages 2024-03-29T11:18:20.347 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:20.347 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 964 k 2024-03-29T11:18:20.347 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 2.8 M 2024-03-29T11:18:20.347 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-03-29T11:18:20.487 INFO:teuthology.orchestra.run.smithi119.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 563 kB/s | 79 kB 00:00 2024-03-29T11:18:20.670 INFO:teuthology.orchestra.run.smithi119.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 2.7 MB/s | 885 kB 00:00 2024-03-29T11:18:20.671 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:20.672 INFO:teuthology.orchestra.run.smithi119.stdout:Total 2.9 MB/s | 964 kB 00:00 2024-03-29T11:18:20.672 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-03-29T11:18:20.692 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-03-29T11:18:20.692 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-03-29T11:18:20.765 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-03-29T11:18:20.765 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-03-29T11:18:20.961 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-03-29T11:18:21.104 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:18:21.169 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:18:21.877 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:18:21.877 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:18:22.159 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:18:22.159 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:22.159 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-03-29T11:18:22.159 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:18:22.159 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:22.159 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-29T11:18:22.448 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-volume 2024-03-29T11:18:23.024 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:04:09 ago on Fri 29 Mar 2024 11:14:14 AM UTC. 2024-03-29T11:18:23.181 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-29T11:18:23.183 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-29T11:18:23.183 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-03-29T11:18:23.183 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-29T11:18:23.183 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-03-29T11:18:23.183 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:18:23.183 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-03-29T11:18:23.184 INFO:teuthology.orchestra.run.smithi119.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:18:23.184 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:23.184 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-03-29T11:18:23.184 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-29T11:18:23.184 INFO:teuthology.orchestra.run.smithi119.stdout:Install 2 Packages 2024-03-29T11:18:23.184 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:23.185 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 332 k 2024-03-29T11:18:23.185 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 1.5 M 2024-03-29T11:18:23.185 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-03-29T11:18:23.308 INFO:teuthology.orchestra.run.smithi119.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 551 kB/s | 69 kB 00:00 2024-03-29T11:18:23.625 INFO:teuthology.orchestra.run.smithi119.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 597 kB/s | 264 kB 00:00 2024-03-29T11:18:23.626 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:23.626 INFO:teuthology.orchestra.run.smithi119.stdout:Total 748 kB/s | 332 kB 00:00 2024-03-29T11:18:23.627 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-03-29T11:18:23.635 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-03-29T11:18:23.636 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-03-29T11:18:23.674 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-03-29T11:18:23.675 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-03-29T11:18:23.821 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-03-29T11:18:24.000 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:18:24.036 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:18:24.052 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:18:24.052 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:24.052 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:18:24.052 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:24.771 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:18:25.092 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:18:25.093 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:25.093 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-03-29T11:18:25.093 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:25.093 INFO:teuthology.orchestra.run.smithi119.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:18:25.093 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:25.093 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-29T11:18:25.439 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install librados-devel 2024-03-29T11:18:26.014 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:04:12 ago on Fri 29 Mar 2024 11:14:14 AM UTC. 2024-03-29T11:18:26.164 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-29T11:18:26.165 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-29T11:18:26.165 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repo Size 2024-03-29T11:18:26.165 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-29T11:18:26.166 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-03-29T11:18:26.166 INFO:teuthology.orchestra.run.smithi119.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:18:26.166 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:26.166 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-03-29T11:18:26.166 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-29T11:18:26.166 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-03-29T11:18:26.166 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:26.166 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 127 k 2024-03-29T11:18:26.166 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 456 k 2024-03-29T11:18:26.166 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-03-29T11:18:26.547 INFO:teuthology.orchestra.run.smithi119.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 333 kB/s | 127 kB 00:00 2024-03-29T11:18:26.548 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:26.548 INFO:teuthology.orchestra.run.smithi119.stdout:Total 331 kB/s | 127 kB 00:00 2024-03-29T11:18:26.548 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-03-29T11:18:26.554 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-03-29T11:18:26.554 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-03-29T11:18:26.610 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-03-29T11:18:26.611 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-03-29T11:18:26.750 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-03-29T11:18:26.856 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:27.281 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:27.576 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:27.576 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:27.576 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-03-29T11:18:27.576 INFO:teuthology.orchestra.run.smithi119.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:27.577 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:27.577 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-29T11:18:27.824 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install libcephfs2 2024-03-29T11:18:28.391 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:04:14 ago on Fri 29 Mar 2024 11:14:14 AM UTC. 2024-03-29T11:18:28.490 INFO:teuthology.orchestra.run.smithi119.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:28.545 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-29T11:18:28.546 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-03-29T11:18:28.546 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-29T11:18:28.614 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install libcephfs-devel 2024-03-29T11:18:29.180 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:04:15 ago on Fri 29 Mar 2024 11:14:14 AM UTC. 2024-03-29T11:18:29.340 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-29T11:18:29.341 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-29T11:18:29.341 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repo Size 2024-03-29T11:18:29.341 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-29T11:18:29.341 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-03-29T11:18:29.341 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:18:29.341 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:29.341 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-03-29T11:18:29.341 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-29T11:18:29.341 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-03-29T11:18:29.341 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:29.342 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 31 k 2024-03-29T11:18:29.342 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 140 k 2024-03-29T11:18:29.342 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-03-29T11:18:29.667 INFO:teuthology.orchestra.run.smithi119.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 96 kB/s | 31 kB 00:00 2024-03-29T11:18:29.668 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:29.668 INFO:teuthology.orchestra.run.smithi119.stdout:Total 95 kB/s | 31 kB 00:00 2024-03-29T11:18:29.668 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-03-29T11:18:29.672 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-03-29T11:18:29.672 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-03-29T11:18:29.699 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-03-29T11:18:29.699 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-03-29T11:18:29.849 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-03-29T11:18:29.940 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:30.299 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:30.555 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:30.555 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:30.555 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-03-29T11:18:30.555 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:30.555 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:30.556 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-29T11:18:30.713 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install librados2 2024-03-29T11:18:31.276 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:04:17 ago on Fri 29 Mar 2024 11:14:14 AM UTC. 2024-03-29T11:18:31.374 INFO:teuthology.orchestra.run.smithi119.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:31.426 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-29T11:18:31.427 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-03-29T11:18:31.427 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-29T11:18:31.479 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install librbd1 2024-03-29T11:18:32.043 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:04:18 ago on Fri 29 Mar 2024 11:14:14 AM UTC. 2024-03-29T11:18:32.141 INFO:teuthology.orchestra.run.smithi119.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:32.192 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-29T11:18:32.193 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-03-29T11:18:32.193 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-29T11:18:32.248 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install python3-rados 2024-03-29T11:18:32.813 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:04:18 ago on Fri 29 Mar 2024 11:14:14 AM UTC. 2024-03-29T11:18:32.911 INFO:teuthology.orchestra.run.smithi119.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:32.963 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-29T11:18:32.964 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-03-29T11:18:32.964 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-29T11:18:33.014 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install python3-rgw 2024-03-29T11:18:33.577 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:04:19 ago on Fri 29 Mar 2024 11:14:14 AM UTC. 2024-03-29T11:18:33.676 INFO:teuthology.orchestra.run.smithi119.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:33.728 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-29T11:18:33.729 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-03-29T11:18:33.729 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-29T11:18:33.779 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install python3-cephfs 2024-03-29T11:18:34.341 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:04:20 ago on Fri 29 Mar 2024 11:14:14 AM UTC. 2024-03-29T11:18:34.439 INFO:teuthology.orchestra.run.smithi119.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:34.490 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-29T11:18:34.491 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-03-29T11:18:34.492 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-29T11:18:34.545 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install python3-rbd 2024-03-29T11:18:35.106 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:04:21 ago on Fri 29 Mar 2024 11:14:14 AM UTC. 2024-03-29T11:18:35.204 INFO:teuthology.orchestra.run.smithi119.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:35.255 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-29T11:18:35.256 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-03-29T11:18:35.256 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-29T11:18:35.307 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install rbd-fuse 2024-03-29T11:18:35.872 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:04:21 ago on Fri 29 Mar 2024 11:14:14 AM UTC. 2024-03-29T11:18:36.024 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-29T11:18:36.025 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-29T11:18:36.025 INFO:teuthology.orchestra.run.smithi119.stdout: Package Architecture Version Repository Size 2024-03-29T11:18:36.025 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-29T11:18:36.025 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-03-29T11:18:36.025 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:18:36.025 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:36.025 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-03-29T11:18:36.025 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-29T11:18:36.025 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-03-29T11:18:36.025 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:36.026 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 85 k 2024-03-29T11:18:36.026 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 231 k 2024-03-29T11:18:36.026 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-03-29T11:18:36.228 INFO:teuthology.orchestra.run.smithi119.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 421 kB/s | 85 kB 00:00 2024-03-29T11:18:36.229 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:36.229 INFO:teuthology.orchestra.run.smithi119.stdout:Total 418 kB/s | 85 kB 00:00 2024-03-29T11:18:36.230 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-03-29T11:18:36.236 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-03-29T11:18:36.236 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-03-29T11:18:36.291 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-03-29T11:18:36.291 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-03-29T11:18:36.434 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-03-29T11:18:36.543 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:36.928 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:37.225 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:37.226 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:37.226 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-03-29T11:18:37.226 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:37.226 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:37.226 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-29T11:18:37.446 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install rbd-mirror 2024-03-29T11:18:38.020 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:04:24 ago on Fri 29 Mar 2024 11:14:14 AM UTC. 2024-03-29T11:18:38.177 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-29T11:18:38.177 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-29T11:18:38.178 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repo Size 2024-03-29T11:18:38.178 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-29T11:18:38.178 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-03-29T11:18:38.178 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:18:38.178 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:38.178 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-03-29T11:18:38.178 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-29T11:18:38.178 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-03-29T11:18:38.178 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:38.178 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 3.1 M 2024-03-29T11:18:38.178 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 13 M 2024-03-29T11:18:38.178 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-03-29T11:18:38.898 INFO:teuthology.orchestra.run.smithi119.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 4.3 MB/s | 3.1 MB 00:00 2024-03-29T11:18:38.899 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:38.899 INFO:teuthology.orchestra.run.smithi119.stdout:Total 4.3 MB/s | 3.1 MB 00:00 2024-03-29T11:18:38.899 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-03-29T11:18:38.908 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-03-29T11:18:38.908 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-03-29T11:18:38.982 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-03-29T11:18:38.982 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-03-29T11:18:39.408 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-03-29T11:18:39.466 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:39.490 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:39.490 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:39.490 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:18:39.490 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:18:39.490 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:18:39.490 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:40.607 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:40.607 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:40.607 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-03-29T11:18:40.607 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:40.607 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:40.608 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-29T11:18:40.853 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install rbd-nbd 2024-03-29T11:18:41.428 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:04:27 ago on Fri 29 Mar 2024 11:14:14 AM UTC. 2024-03-29T11:18:41.577 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-03-29T11:18:41.577 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-29T11:18:41.577 INFO:teuthology.orchestra.run.smithi119.stdout: Package Architecture Version Repository Size 2024-03-29T11:18:41.577 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-29T11:18:41.577 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-03-29T11:18:41.578 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:18:41.578 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:41.578 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-03-29T11:18:41.578 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-03-29T11:18:41.578 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-03-29T11:18:41.578 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:41.578 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 171 k 2024-03-29T11:18:41.579 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 494 k 2024-03-29T11:18:41.579 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-03-29T11:18:41.810 INFO:teuthology.orchestra.run.smithi119.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 743 kB/s | 171 kB 00:00 2024-03-29T11:18:41.810 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:41.810 INFO:teuthology.orchestra.run.smithi119.stdout:Total 739 kB/s | 171 kB 00:00 2024-03-29T11:18:41.810 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-03-29T11:18:41.818 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-03-29T11:18:41.818 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-03-29T11:18:41.874 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-03-29T11:18:41.874 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-03-29T11:18:42.021 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-03-29T11:18:42.126 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:42.522 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:42.822 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:42.823 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:42.823 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-03-29T11:18:42.823 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:42.823 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:42.823 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-03-29T11:18:43.055 DEBUG:teuthology.parallel:result is None 2024-03-29T11:18:43.055 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:18:43.273 DEBUG:teuthology.orchestra.run.smithi104:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:18:43.298 INFO:teuthology.orchestra.run.smithi104.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:18:43.299 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:18:43.299 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:18:43.301 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:18:43.437 DEBUG:teuthology.orchestra.run.smithi119:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:18:43.458 INFO:teuthology.orchestra.run.smithi119.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:18:43.458 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:18:43.458 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:18:43.460 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T11:18:43.460 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-29T11:18:43.460 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:18:43.492 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-29T11:18:43.492 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:18:43.525 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T11:18:43.525 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-29T11:18:43.525 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:18:43.566 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:18:43.635 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-29T11:18:43.635 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:18:43.661 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:18:43.734 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T11:18:43.735 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-29T11:18:43.735 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:18:43.762 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:18:43.829 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-29T11:18:43.829 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:18:43.856 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:18:43.922 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T11:18:43.923 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-29T11:18:43.923 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:18:43.950 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:18:44.017 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-29T11:18:44.017 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:18:44.043 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:18:44.109 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T11:18:44.198 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T11:18:44.198 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 777 /var/log/ceph 2024-03-29T11:18:44.200 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 777 /var/log/ceph 2024-03-29T11:18:44.226 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T11:18:44.227 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:18:44.268 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:18:44.295 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T11:18:44.295 DEBUG:teuthology.orchestra.run.smithi104:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:18:44.337 DEBUG:teuthology.orchestra.run.smithi119:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:18:44.373 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T11:18:44.373 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-29T11:18:44.374 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622681', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7622681', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8152, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi104.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMapDgIsdLYjW2H7cUSLyAA2wd86CeL/sKuJHvyFFC9a+OL//CvJkSpgxOVDA9BOb4KTT7tX67mxUCkvnffYm4g=', 'smithi119.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM6RWsxLIJXsXGfKg2EOoww8oIOk8ov7+dbvExTnnCoD8eyyxNhyxmQFEDeoq7WrTYD4l6S+hXD66rx7Qe4rA34='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-26_05:45:44', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T11:18:44.374 DEBUG:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:18:44.404 DEBUG:teuthology.orchestra.run.smithi119:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:18:44.418 DEBUG:teuthology.orchestra.run.smithi104:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:18:44.459 DEBUG:teuthology.orchestra.run.smithi119:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:18:44.485 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-29T11:18:44.485 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:18:44.538 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:18:44.538 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_1 2024-03-29T11:18:44.592 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:18:44.592 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:18:44.592 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-29T11:18:44.593 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:18:44.593 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:18:44.593 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-03-29 11:15:41.859607798 +0000 2024-03-29T11:18:44.593 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-03-29 11:13:09.013515149 +0000 2024-03-29T11:18:44.593 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-03-29 11:13:09.013515149 +0000 2024-03-29T11:18:44.593 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-03-29 11:13:09.013515149 +0000 2024-03-29T11:18:44.593 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:18:44.657 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-03-29T11:18:44.657 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-03-29T11:18:44.658 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000244839 s, 2.1 MB/s 2024-03-29T11:18:44.659 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:18:44.713 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_2 2024-03-29T11:18:44.768 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:18:44.768 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:18:44.768 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-29T11:18:44.768 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:18:44.769 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:18:44.769 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-03-29 11:15:41.860607786 +0000 2024-03-29T11:18:44.769 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-03-29 11:13:09.393510817 +0000 2024-03-29T11:18:44.769 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-03-29 11:13:09.393510817 +0000 2024-03-29T11:18:44.769 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-03-29 11:13:09.393510817 +0000 2024-03-29T11:18:44.769 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:18:44.834 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-03-29T11:18:44.834 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-03-29T11:18:44.834 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000209297 s, 2.4 MB/s 2024-03-29T11:18:44.835 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:18:44.891 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_3 2024-03-29T11:18:44.946 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:18:44.946 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:18:44.946 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-29T11:18:44.947 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:18:44.947 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:18:44.947 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-03-29 11:15:41.860607786 +0000 2024-03-29T11:18:44.947 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-03-29 11:13:09.742506839 +0000 2024-03-29T11:18:44.947 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-03-29 11:13:09.742506839 +0000 2024-03-29T11:18:44.947 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-03-29 11:13:09.742506839 +0000 2024-03-29T11:18:44.947 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:18:45.011 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-03-29T11:18:45.011 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-03-29T11:18:45.012 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000215481 s, 2.4 MB/s 2024-03-29T11:18:45.013 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:18:45.069 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_4 2024-03-29T11:18:45.123 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:18:45.123 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:18:45.123 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-29T11:18:45.124 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:18:45.124 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:18:45.124 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-03-29 11:15:41.860607786 +0000 2024-03-29T11:18:45.124 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-03-29 11:13:10.091502861 +0000 2024-03-29T11:18:45.124 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-03-29 11:13:10.091502861 +0000 2024-03-29T11:18:45.124 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-03-29 11:13:10.091502861 +0000 2024-03-29T11:18:45.124 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:18:45.189 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-03-29T11:18:45.189 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-03-29T11:18:45.189 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000203358 s, 2.5 MB/s 2024-03-29T11:18:45.190 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:18:45.246 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:18:45.246 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-29T11:18:45.246 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:18:45.263 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:18:45.263 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_1 2024-03-29T11:18:45.320 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:18:45.320 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:18:45.320 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-29T11:18:45.321 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:18:45.321 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:18:45.321 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-03-29 11:18:39.765957719 +0000 2024-03-29T11:18:45.321 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-03-29 11:13:03.524022268 +0000 2024-03-29T11:18:45.321 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-03-29 11:13:03.524022268 +0000 2024-03-29T11:18:45.321 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: 2024-03-29 11:13:03.524022268 +0000 2024-03-29T11:18:45.321 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:18:45.388 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-03-29T11:18:45.388 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-03-29T11:18:45.389 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.00021446 s, 2.4 MB/s 2024-03-29T11:18:45.390 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:18:45.447 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_2 2024-03-29T11:18:45.502 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:18:45.502 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:18:45.502 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-29T11:18:45.503 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:18:45.503 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:18:45.503 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-03-29 11:18:39.765957719 +0000 2024-03-29T11:18:45.503 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-03-29 11:13:03.867013886 +0000 2024-03-29T11:18:45.503 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-03-29 11:13:03.867013886 +0000 2024-03-29T11:18:45.503 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: 2024-03-29 11:13:03.867013886 +0000 2024-03-29T11:18:45.503 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:18:45.569 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-03-29T11:18:45.569 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-03-29T11:18:45.569 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000223801 s, 2.3 MB/s 2024-03-29T11:18:45.570 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:18:45.627 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_3 2024-03-29T11:18:45.682 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:18:45.682 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:18:45.682 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-29T11:18:45.682 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:18:45.682 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:18:45.682 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-03-29 11:18:39.765957719 +0000 2024-03-29T11:18:45.682 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-03-29 11:13:04.209005529 +0000 2024-03-29T11:18:45.682 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-03-29 11:13:04.209005529 +0000 2024-03-29T11:18:45.683 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: 2024-03-29 11:13:04.209005529 +0000 2024-03-29T11:18:45.683 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:18:45.747 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-03-29T11:18:45.747 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-03-29T11:18:45.748 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000195229 s, 2.6 MB/s 2024-03-29T11:18:45.749 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:18:45.805 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_4 2024-03-29T11:18:45.860 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:18:45.860 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:18:45.860 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-29T11:18:45.860 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:18:45.860 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:18:45.860 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-03-29 11:18:39.766957694 +0000 2024-03-29T11:18:45.860 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-03-29 11:13:04.563996854 +0000 2024-03-29T11:18:45.860 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-03-29 11:13:04.563996854 +0000 2024-03-29T11:18:45.860 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: 2024-03-29 11:13:04.563996854 +0000 2024-03-29T11:18:45.861 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:18:45.926 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-03-29T11:18:45.926 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-03-29T11:18:45.926 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000190282 s, 2.7 MB/s 2024-03-29T11:18:45.927 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:18:45.983 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:18:45.983 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi104.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi119.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:18:45.983 INFO:tasks.ceph:Generating config... 2024-03-29T11:18:45.985 INFO:tasks.ceph:[client] debug monc = 20 2024-03-29T11:18:45.985 INFO:tasks.ceph:[client] debug ms = 1 2024-03-29T11:18:45.985 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-29T11:18:45.985 INFO:tasks.ceph:[client] debug rados = 20 2024-03-29T11:18:45.985 INFO:tasks.ceph:[global] lockdep = True 2024-03-29T11:18:45.985 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-29T11:18:45.985 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-29T11:18:45.985 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-29T11:18:45.985 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-29T11:18:45.985 INFO:tasks.ceph:[global] ms type = async 2024-03-29T11:18:45.985 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T11:18:45.985 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T11:18:45.985 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-29T11:18:45.985 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-29T11:18:45.986 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-29T11:18:45.986 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-29T11:18:45.986 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-29T11:18:45.986 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-29T11:18:45.986 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-29T11:18:45.986 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-29T11:18:45.986 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-29T11:18:45.986 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-29T11:18:45.986 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T11:18:45.986 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2024-03-29T11:18:45.986 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T11:18:45.986 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T11:18:45.986 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-29T11:18:45.986 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T11:18:45.986 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T11:18:45.987 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T11:18:45.987 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-29T11:18:45.987 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-29T11:18:45.987 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T11:18:45.987 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T11:18:45.987 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T11:18:45.987 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T11:18:45.987 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-29T11:18:45.987 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-29T11:18:45.987 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-29T11:18:45.987 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-29T11:18:45.987 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-29T11:18:45.987 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T11:18:45.987 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-29T11:18:45.987 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-29T11:18:45.987 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T11:18:45.988 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-29T11:18:45.988 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-29T11:18:45.988 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T11:18:45.988 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T11:18:45.988 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T11:18:46.050 INFO:teuthology.orchestra.run.smithi104.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T11:18:46.053 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T11:18:46.098 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:18:46.125 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.104:6789]'), ('mon.c', '[v1:172.21.15.104:6790]'), ('mon.b', '[v1:172.21.15.119:6789]')] 2024-03-29T11:18:46.126 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.104:6789],[v1:172.21.15.104:6790],[v1:172.21.15.119:6789]', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-29T11:18:46.127 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-29T11:18:46.127 DEBUG:teuthology.orchestra.run.smithi104:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:18:46.181 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.104:6789]' --addv c '[v1:172.21.15.104:6790]' --addv b '[v1:172.21.15.119:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:46.265 INFO:teuthology.orchestra.run.smithi104.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:46.266 INFO:teuthology.orchestra.run.smithi104.stdout:monmaptool: generated fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:18:46.266 INFO:teuthology.orchestra.run.smithi104.stdout:setting min_mon_release = quincy 2024-03-29T11:18:46.266 INFO:teuthology.orchestra.run.smithi104.stdout:epoch 0 2024-03-29T11:18:46.266 INFO:teuthology.orchestra.run.smithi104.stdout:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:18:46.266 INFO:teuthology.orchestra.run.smithi104.stdout:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:18:46.266 INFO:teuthology.orchestra.run.smithi104.stdout:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:18:46.266 INFO:teuthology.orchestra.run.smithi104.stdout:min_mon_release 17 (quincy) 2024-03-29T11:18:46.266 INFO:teuthology.orchestra.run.smithi104.stdout:election_strategy: 1 2024-03-29T11:18:46.266 INFO:teuthology.orchestra.run.smithi104.stdout:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:18:46.267 INFO:teuthology.orchestra.run.smithi104.stdout:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:18:46.267 INFO:teuthology.orchestra.run.smithi104.stdout:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:18:46.267 INFO:teuthology.orchestra.run.smithi104.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-29T11:18:46.268 DEBUG:teuthology.orchestra.run.smithi104:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:18:46.282 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID da26e8fd-d28e-492a-aadb-08e647a06b62... 2024-03-29T11:18:46.283 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:18:46.325 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:18:46.373 INFO:teuthology.orchestra.run.smithi104.stdout:[global] 2024-03-29T11:18:46.373 INFO:teuthology.orchestra.run.smithi104.stdout: chdir = "" 2024-03-29T11:18:46.373 INFO:teuthology.orchestra.run.smithi104.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:18:46.373 INFO:teuthology.orchestra.run.smithi104.stdout: auth supported = cephx 2024-03-29T11:18:46.373 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:18:46.373 INFO:teuthology.orchestra.run.smithi104.stdout: filestore xattr use omap = true 2024-03-29T11:18:46.373 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:18:46.373 INFO:teuthology.orchestra.run.smithi104.stdout: mon clock drift allowed = 1.000 2024-03-29T11:18:46.373 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:18:46.373 INFO:teuthology.orchestra.run.smithi104.stdout: osd crush chooseleaf type = 0 2024-03-29T11:18:46.373 INFO:teuthology.orchestra.run.smithi104.stdout: auth debug = true 2024-03-29T11:18:46.374 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:18:46.374 INFO:teuthology.orchestra.run.smithi104.stdout: ms die on old message = true 2024-03-29T11:18:46.374 INFO:teuthology.orchestra.run.smithi104.stdout: ms die on bug = true 2024-03-29T11:18:46.374 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:18:46.374 INFO:teuthology.orchestra.run.smithi104.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:18:46.374 INFO:teuthology.orchestra.run.smithi104.stdout: mon pg warn max object skew = 0 2024-03-29T11:18:46.374 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:18:46.374 INFO:teuthology.orchestra.run.smithi104.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:18:46.374 INFO:teuthology.orchestra.run.smithi104.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:18:46.374 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:18:46.374 INFO:teuthology.orchestra.run.smithi104.stdout: osd pool default size = 2 2024-03-29T11:18:46.374 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:18:46.374 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd allow primary affinity = true 2024-03-29T11:18:46.375 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd allow pg remap = true 2024-03-29T11:18:46.375 INFO:teuthology.orchestra.run.smithi104.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:18:46.375 INFO:teuthology.orchestra.run.smithi104.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:18:46.375 INFO:teuthology.orchestra.run.smithi104.stdout: mon warn on no sortbitwise = false 2024-03-29T11:18:46.375 INFO:teuthology.orchestra.run.smithi104.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:18:46.375 INFO:teuthology.orchestra.run.smithi104.stdout: mon warn on too few osds = false 2024-03-29T11:18:46.375 INFO:teuthology.orchestra.run.smithi104.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:18:46.375 INFO:teuthology.orchestra.run.smithi104.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:18:46.375 INFO:teuthology.orchestra.run.smithi104.stdout: mon_allow_pool_size_one = true 2024-03-29T11:18:46.376 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:18:46.376 INFO:teuthology.orchestra.run.smithi104.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:18:46.376 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:18:46.376 INFO:teuthology.orchestra.run.smithi104.stdout: osd default data pool replay window = 5 2024-03-29T11:18:46.376 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:18:46.376 INFO:teuthology.orchestra.run.smithi104.stdout: mon allow pool delete = true 2024-03-29T11:18:46.376 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:18:46.376 INFO:teuthology.orchestra.run.smithi104.stdout: mon cluster log file level = debug 2024-03-29T11:18:46.376 INFO:teuthology.orchestra.run.smithi104.stdout: debug asserts on shutdown = true 2024-03-29T11:18:46.377 INFO:teuthology.orchestra.run.smithi104.stdout: mon health detail to clog = false 2024-03-29T11:18:46.377 INFO:teuthology.orchestra.run.smithi104.stdout: mon host = "[v1:172.21.15.104:6789],[v1:172.21.15.104:6790],[v1:172.21.15.119:6789]" 2024-03-29T11:18:46.377 INFO:teuthology.orchestra.run.smithi104.stdout: lockdep = True 2024-03-29T11:18:46.377 INFO:teuthology.orchestra.run.smithi104.stdout: mon client directed command retry = 5 2024-03-29T11:18:46.377 INFO:teuthology.orchestra.run.smithi104.stdout: mon election default strategy = 1 2024-03-29T11:18:46.377 INFO:teuthology.orchestra.run.smithi104.stdout: ms bind msgr2 = False 2024-03-29T11:18:46.377 INFO:teuthology.orchestra.run.smithi104.stdout: ms inject socket failures = 5000 2024-03-29T11:18:46.378 INFO:teuthology.orchestra.run.smithi104.stdout: ms type = async 2024-03-29T11:18:46.378 INFO:teuthology.orchestra.run.smithi104.stdout: fsid = da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:18:46.378 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:18:46.378 INFO:teuthology.orchestra.run.smithi104.stdout:[osd] 2024-03-29T11:18:46.378 INFO:teuthology.orchestra.run.smithi104.stdout: osd journal size = 100 2024-03-29T11:18:46.378 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:18:46.378 INFO:teuthology.orchestra.run.smithi104.stdout: osd scrub load threshold = 5.0 2024-03-29T11:18:46.378 INFO:teuthology.orchestra.run.smithi104.stdout: osd scrub max interval = 600 2024-03-29T11:18:46.379 INFO:teuthology.orchestra.run.smithi104.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:18:46.379 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:18:46.379 INFO:teuthology.orchestra.run.smithi104.stdout: osd recover clone overlap = true 2024-03-29T11:18:46.379 INFO:teuthology.orchestra.run.smithi104.stdout: osd recovery max chunk = 1048576 2024-03-29T11:18:46.379 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:18:46.379 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug shutdown = true 2024-03-29T11:18:46.379 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug op order = true 2024-03-29T11:18:46.379 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug verify stray on activate = true 2024-03-29T11:18:46.379 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug trim objects = true 2024-03-29T11:18:46.379 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:18:46.379 INFO:teuthology.orchestra.run.smithi104.stdout: osd open classes on start = true 2024-03-29T11:18:46.379 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug pg log writeout = true 2024-03-29T11:18:46.379 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:18:46.379 INFO:teuthology.orchestra.run.smithi104.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:18:46.379 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:18:46.380 INFO:teuthology.orchestra.run.smithi104.stdout: osd map max advance = 10 2024-03-29T11:18:46.380 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:18:46.380 INFO:teuthology.orchestra.run.smithi104.stdout: journal zero on create = true 2024-03-29T11:18:46.380 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:18:46.380 INFO:teuthology.orchestra.run.smithi104.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:18:46.380 INFO:teuthology.orchestra.run.smithi104.stdout: filestore apply finisher threads = 3 2024-03-29T11:18:46.380 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:18:46.380 INFO:teuthology.orchestra.run.smithi104.stdout: bdev debug aio = true 2024-03-29T11:18:46.380 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug misdirected ops = true 2024-03-29T11:18:46.380 INFO:teuthology.orchestra.run.smithi104.stdout: bluestore block size = 96636764160 2024-03-29T11:18:46.380 INFO:teuthology.orchestra.run.smithi104.stdout: bluestore compression algorithm = zstd 2024-03-29T11:18:46.380 INFO:teuthology.orchestra.run.smithi104.stdout: bluestore compression mode = aggressive 2024-03-29T11:18:46.380 INFO:teuthology.orchestra.run.smithi104.stdout: bluestore fsck on mount = True 2024-03-29T11:18:46.380 INFO:teuthology.orchestra.run.smithi104.stdout: bluestore zero block detection = True 2024-03-29T11:18:46.381 INFO:teuthology.orchestra.run.smithi104.stdout: debug bluefs = 20 2024-03-29T11:18:46.381 INFO:teuthology.orchestra.run.smithi104.stdout: debug bluestore = 20 2024-03-29T11:18:46.381 INFO:teuthology.orchestra.run.smithi104.stdout: debug monc = 20 2024-03-29T11:18:46.381 INFO:teuthology.orchestra.run.smithi104.stdout: debug ms = 1 2024-03-29T11:18:46.381 INFO:teuthology.orchestra.run.smithi104.stdout: debug osd = 20 2024-03-29T11:18:46.381 INFO:teuthology.orchestra.run.smithi104.stdout: debug rocksdb = 10 2024-03-29T11:18:46.381 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:18:46.381 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd full ratio = 0.9 2024-03-29T11:18:46.381 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:18:46.381 INFO:teuthology.orchestra.run.smithi104.stdout: osd class default list = * 2024-03-29T11:18:46.381 INFO:teuthology.orchestra.run.smithi104.stdout: osd class load list = * 2024-03-29T11:18:46.381 INFO:teuthology.orchestra.run.smithi104.stdout: osd client watch timeout = 120 2024-03-29T11:18:46.381 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug verify cached snaps = True 2024-03-29T11:18:46.381 INFO:teuthology.orchestra.run.smithi104.stdout: osd debug verify missing on start = True 2024-03-29T11:18:46.382 INFO:teuthology.orchestra.run.smithi104.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:18:46.382 INFO:teuthology.orchestra.run.smithi104.stdout: osd mclock override recovery settings = True 2024-03-29T11:18:46.382 INFO:teuthology.orchestra.run.smithi104.stdout: osd objectstore = bluestore 2024-03-29T11:18:46.382 INFO:teuthology.orchestra.run.smithi104.stdout: osd op queue = debug_random 2024-03-29T11:18:46.382 INFO:teuthology.orchestra.run.smithi104.stdout: osd op queue cut off = debug_random 2024-03-29T11:18:46.382 INFO:teuthology.orchestra.run.smithi104.stdout: osd shutdown pgref assert = True 2024-03-29T11:18:46.382 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:18:46.382 INFO:teuthology.orchestra.run.smithi104.stdout:[mgr] 2024-03-29T11:18:46.382 INFO:teuthology.orchestra.run.smithi104.stdout: debug ms = 1 2024-03-29T11:18:46.382 INFO:teuthology.orchestra.run.smithi104.stdout: debug mgr = 20 2024-03-29T11:18:46.382 INFO:teuthology.orchestra.run.smithi104.stdout: debug mon = 20 2024-03-29T11:18:46.382 INFO:teuthology.orchestra.run.smithi104.stdout: debug auth = 20 2024-03-29T11:18:46.382 INFO:teuthology.orchestra.run.smithi104.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:18:46.383 INFO:teuthology.orchestra.run.smithi104.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:18:46.383 INFO:teuthology.orchestra.run.smithi104.stdout: mgr/telemetry/nag = false 2024-03-29T11:18:46.383 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:18:46.383 INFO:teuthology.orchestra.run.smithi104.stdout:[mon] 2024-03-29T11:18:46.383 INFO:teuthology.orchestra.run.smithi104.stdout: debug ms = 1 2024-03-29T11:18:46.383 INFO:teuthology.orchestra.run.smithi104.stdout: debug mon = 20 2024-03-29T11:18:46.383 INFO:teuthology.orchestra.run.smithi104.stdout: debug paxos = 20 2024-03-29T11:18:46.383 INFO:teuthology.orchestra.run.smithi104.stdout: debug auth = 20 2024-03-29T11:18:46.383 INFO:teuthology.orchestra.run.smithi104.stdout: mon data avail warn = 5 2024-03-29T11:18:46.383 INFO:teuthology.orchestra.run.smithi104.stdout: mon mgr mkfs grace = 240 2024-03-29T11:18:46.383 INFO:teuthology.orchestra.run.smithi104.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:18:46.383 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd reporter subtree level = osd 2024-03-29T11:18:46.383 INFO:teuthology.orchestra.run.smithi104.stdout: mon osd prime pg temp = true 2024-03-29T11:18:46.384 INFO:teuthology.orchestra.run.smithi104.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:18:46.384 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:18:46.384 INFO:teuthology.orchestra.run.smithi104.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:18:46.384 INFO:teuthology.orchestra.run.smithi104.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:18:46.384 INFO:teuthology.orchestra.run.smithi104.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:18:46.384 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:18:46.384 INFO:teuthology.orchestra.run.smithi104.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:18:46.384 INFO:teuthology.orchestra.run.smithi104.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:18:46.384 INFO:teuthology.orchestra.run.smithi104.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:18:46.384 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:18:46.384 INFO:teuthology.orchestra.run.smithi104.stdout: # 1m isn't quite enough 2024-03-29T11:18:46.384 INFO:teuthology.orchestra.run.smithi104.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:18:46.384 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:18:46.384 INFO:teuthology.orchestra.run.smithi104.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:18:46.384 INFO:teuthology.orchestra.run.smithi104.stdout: mon min osdmap epochs = 50 2024-03-29T11:18:46.385 INFO:teuthology.orchestra.run.smithi104.stdout: mon osdmap full prune interval = 2 2024-03-29T11:18:46.385 INFO:teuthology.orchestra.run.smithi104.stdout: mon osdmap full prune min = 15 2024-03-29T11:18:46.385 INFO:teuthology.orchestra.run.smithi104.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:18:46.385 INFO:teuthology.orchestra.run.smithi104.stdout: mon scrub interval = 300 2024-03-29T11:18:46.385 INFO:teuthology.orchestra.run.smithi104.stdout: mon warn on pool no app = False 2024-03-29T11:18:46.385 INFO:teuthology.orchestra.run.smithi104.stdout: paxos service trim min = 10 2024-03-29T11:18:46.385 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:18:46.385 INFO:teuthology.orchestra.run.smithi104.stdout:[client] 2024-03-29T11:18:46.385 INFO:teuthology.orchestra.run.smithi104.stdout: rgw cache enabled = true 2024-03-29T11:18:46.385 INFO:teuthology.orchestra.run.smithi104.stdout: rgw enable ops log = true 2024-03-29T11:18:46.385 INFO:teuthology.orchestra.run.smithi104.stdout: rgw enable usage log = true 2024-03-29T11:18:46.385 INFO:teuthology.orchestra.run.smithi104.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:18:46.385 INFO:teuthology.orchestra.run.smithi104.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:18:46.385 INFO:teuthology.orchestra.run.smithi104.stdout: debug monc = 20 2024-03-29T11:18:46.386 INFO:teuthology.orchestra.run.smithi104.stdout: debug ms = 1 2024-03-29T11:18:46.386 INFO:teuthology.orchestra.run.smithi104.stdout: debug objecter = 20 2024-03-29T11:18:46.386 INFO:teuthology.orchestra.run.smithi104.stdout: debug rados = 20 2024-03-29T11:18:46.386 INFO:teuthology.orchestra.run.smithi104.stdout:[mon.a] 2024-03-29T11:18:46.386 INFO:teuthology.orchestra.run.smithi104.stdout:[mon.c] 2024-03-29T11:18:46.386 INFO:teuthology.orchestra.run.smithi104.stdout:[mon.b] 2024-03-29T11:18:46.386 INFO:teuthology.orchestra.run.smithi119.stdout:[global] 2024-03-29T11:18:46.387 INFO:teuthology.orchestra.run.smithi119.stdout: chdir = "" 2024-03-29T11:18:46.387 INFO:teuthology.orchestra.run.smithi119.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:18:46.387 INFO:teuthology.orchestra.run.smithi119.stdout: auth supported = cephx 2024-03-29T11:18:46.387 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:46.387 INFO:teuthology.orchestra.run.smithi119.stdout: filestore xattr use omap = true 2024-03-29T11:18:46.387 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:46.387 INFO:teuthology.orchestra.run.smithi119.stdout: mon clock drift allowed = 1.000 2024-03-29T11:18:46.387 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:46.387 INFO:teuthology.orchestra.run.smithi119.stdout: osd crush chooseleaf type = 0 2024-03-29T11:18:46.387 INFO:teuthology.orchestra.run.smithi119.stdout: auth debug = true 2024-03-29T11:18:46.387 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:46.387 INFO:teuthology.orchestra.run.smithi119.stdout: ms die on old message = true 2024-03-29T11:18:46.387 INFO:teuthology.orchestra.run.smithi119.stdout: ms die on bug = true 2024-03-29T11:18:46.388 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:46.388 INFO:teuthology.orchestra.run.smithi119.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:18:46.388 INFO:teuthology.orchestra.run.smithi119.stdout: mon pg warn max object skew = 0 2024-03-29T11:18:46.388 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:46.388 INFO:teuthology.orchestra.run.smithi119.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:18:46.388 INFO:teuthology.orchestra.run.smithi119.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:18:46.388 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:46.388 INFO:teuthology.orchestra.run.smithi119.stdout: osd pool default size = 2 2024-03-29T11:18:46.388 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:46.388 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd allow primary affinity = true 2024-03-29T11:18:46.388 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd allow pg remap = true 2024-03-29T11:18:46.388 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:18:46.388 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:18:46.388 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on no sortbitwise = false 2024-03-29T11:18:46.389 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:18:46.389 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on too few osds = false 2024-03-29T11:18:46.389 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:18:46.389 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:18:46.389 INFO:teuthology.orchestra.run.smithi119.stdout: mon_allow_pool_size_one = true 2024-03-29T11:18:46.389 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:46.389 INFO:teuthology.orchestra.run.smithi119.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:18:46.389 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:46.389 INFO:teuthology.orchestra.run.smithi119.stdout: osd default data pool replay window = 5 2024-03-29T11:18:46.389 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:46.389 INFO:teuthology.orchestra.run.smithi119.stdout: mon allow pool delete = true 2024-03-29T11:18:46.389 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:46.389 INFO:teuthology.orchestra.run.smithi119.stdout: mon cluster log file level = debug 2024-03-29T11:18:46.389 INFO:teuthology.orchestra.run.smithi119.stdout: debug asserts on shutdown = true 2024-03-29T11:18:46.390 INFO:teuthology.orchestra.run.smithi119.stdout: mon health detail to clog = false 2024-03-29T11:18:46.390 INFO:teuthology.orchestra.run.smithi119.stdout: mon host = "[v1:172.21.15.104:6789],[v1:172.21.15.104:6790],[v1:172.21.15.119:6789]" 2024-03-29T11:18:46.390 INFO:teuthology.orchestra.run.smithi119.stdout: lockdep = True 2024-03-29T11:18:46.390 INFO:teuthology.orchestra.run.smithi119.stdout: mon client directed command retry = 5 2024-03-29T11:18:46.390 INFO:teuthology.orchestra.run.smithi119.stdout: mon election default strategy = 1 2024-03-29T11:18:46.390 INFO:teuthology.orchestra.run.smithi119.stdout: ms bind msgr2 = False 2024-03-29T11:18:46.390 INFO:teuthology.orchestra.run.smithi119.stdout: ms inject socket failures = 5000 2024-03-29T11:18:46.390 INFO:teuthology.orchestra.run.smithi119.stdout: ms type = async 2024-03-29T11:18:46.390 INFO:teuthology.orchestra.run.smithi119.stdout: fsid = da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:18:46.390 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:46.390 INFO:teuthology.orchestra.run.smithi119.stdout:[osd] 2024-03-29T11:18:46.390 INFO:teuthology.orchestra.run.smithi119.stdout: osd journal size = 100 2024-03-29T11:18:46.391 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:46.391 INFO:teuthology.orchestra.run.smithi119.stdout: osd scrub load threshold = 5.0 2024-03-29T11:18:46.391 INFO:teuthology.orchestra.run.smithi119.stdout: osd scrub max interval = 600 2024-03-29T11:18:46.391 INFO:teuthology.orchestra.run.smithi119.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:18:46.391 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:46.391 INFO:teuthology.orchestra.run.smithi119.stdout: osd recover clone overlap = true 2024-03-29T11:18:46.391 INFO:teuthology.orchestra.run.smithi119.stdout: osd recovery max chunk = 1048576 2024-03-29T11:18:46.391 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:46.391 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug shutdown = true 2024-03-29T11:18:46.391 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug op order = true 2024-03-29T11:18:46.391 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug verify stray on activate = true 2024-03-29T11:18:46.391 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug trim objects = true 2024-03-29T11:18:46.391 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:46.391 INFO:teuthology.orchestra.run.smithi119.stdout: osd open classes on start = true 2024-03-29T11:18:46.391 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug pg log writeout = true 2024-03-29T11:18:46.392 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:46.392 INFO:teuthology.orchestra.run.smithi119.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:18:46.392 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:46.392 INFO:teuthology.orchestra.run.smithi119.stdout: osd map max advance = 10 2024-03-29T11:18:46.392 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:46.392 INFO:teuthology.orchestra.run.smithi119.stdout: journal zero on create = true 2024-03-29T11:18:46.392 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:46.392 INFO:teuthology.orchestra.run.smithi119.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:18:46.392 INFO:teuthology.orchestra.run.smithi119.stdout: filestore apply finisher threads = 3 2024-03-29T11:18:46.392 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:46.392 INFO:teuthology.orchestra.run.smithi119.stdout: bdev debug aio = true 2024-03-29T11:18:46.392 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug misdirected ops = true 2024-03-29T11:18:46.392 INFO:teuthology.orchestra.run.smithi119.stdout: bluestore block size = 96636764160 2024-03-29T11:18:46.392 INFO:teuthology.orchestra.run.smithi119.stdout: bluestore compression algorithm = zstd 2024-03-29T11:18:46.393 INFO:teuthology.orchestra.run.smithi119.stdout: bluestore compression mode = aggressive 2024-03-29T11:18:46.393 INFO:teuthology.orchestra.run.smithi119.stdout: bluestore fsck on mount = True 2024-03-29T11:18:46.393 INFO:teuthology.orchestra.run.smithi119.stdout: bluestore zero block detection = True 2024-03-29T11:18:46.393 INFO:teuthology.orchestra.run.smithi119.stdout: debug bluefs = 20 2024-03-29T11:18:46.393 INFO:teuthology.orchestra.run.smithi119.stdout: debug bluestore = 20 2024-03-29T11:18:46.393 INFO:teuthology.orchestra.run.smithi119.stdout: debug monc = 20 2024-03-29T11:18:46.393 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2024-03-29T11:18:46.393 INFO:teuthology.orchestra.run.smithi119.stdout: debug osd = 20 2024-03-29T11:18:46.393 INFO:teuthology.orchestra.run.smithi119.stdout: debug rocksdb = 10 2024-03-29T11:18:46.393 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:18:46.393 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd full ratio = 0.9 2024-03-29T11:18:46.393 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:18:46.393 INFO:teuthology.orchestra.run.smithi119.stdout: osd class default list = * 2024-03-29T11:18:46.393 INFO:teuthology.orchestra.run.smithi119.stdout: osd class load list = * 2024-03-29T11:18:46.394 INFO:teuthology.orchestra.run.smithi119.stdout: osd client watch timeout = 120 2024-03-29T11:18:46.394 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug verify cached snaps = True 2024-03-29T11:18:46.394 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug verify missing on start = True 2024-03-29T11:18:46.394 INFO:teuthology.orchestra.run.smithi119.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:18:46.394 INFO:teuthology.orchestra.run.smithi119.stdout: osd mclock override recovery settings = True 2024-03-29T11:18:46.394 INFO:teuthology.orchestra.run.smithi119.stdout: osd objectstore = bluestore 2024-03-29T11:18:46.394 INFO:teuthology.orchestra.run.smithi119.stdout: osd op queue = debug_random 2024-03-29T11:18:46.394 INFO:teuthology.orchestra.run.smithi119.stdout: osd op queue cut off = debug_random 2024-03-29T11:18:46.394 INFO:teuthology.orchestra.run.smithi119.stdout: osd shutdown pgref assert = True 2024-03-29T11:18:46.394 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:46.394 INFO:teuthology.orchestra.run.smithi119.stdout:[mgr] 2024-03-29T11:18:46.394 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2024-03-29T11:18:46.394 INFO:teuthology.orchestra.run.smithi119.stdout: debug mgr = 20 2024-03-29T11:18:46.394 INFO:teuthology.orchestra.run.smithi119.stdout: debug mon = 20 2024-03-29T11:18:46.394 INFO:teuthology.orchestra.run.smithi119.stdout: debug auth = 20 2024-03-29T11:18:46.395 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:18:46.395 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:18:46.395 INFO:teuthology.orchestra.run.smithi119.stdout: mgr/telemetry/nag = false 2024-03-29T11:18:46.395 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:46.395 INFO:teuthology.orchestra.run.smithi119.stdout:[mon] 2024-03-29T11:18:46.395 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2024-03-29T11:18:46.395 INFO:teuthology.orchestra.run.smithi119.stdout: debug mon = 20 2024-03-29T11:18:46.395 INFO:teuthology.orchestra.run.smithi119.stdout: debug paxos = 20 2024-03-29T11:18:46.395 INFO:teuthology.orchestra.run.smithi119.stdout: debug auth = 20 2024-03-29T11:18:46.395 INFO:teuthology.orchestra.run.smithi119.stdout: mon data avail warn = 5 2024-03-29T11:18:46.395 INFO:teuthology.orchestra.run.smithi119.stdout: mon mgr mkfs grace = 240 2024-03-29T11:18:46.395 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:18:46.395 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd reporter subtree level = osd 2024-03-29T11:18:46.395 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd prime pg temp = true 2024-03-29T11:18:46.395 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:18:46.396 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:46.396 INFO:teuthology.orchestra.run.smithi119.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:18:46.396 INFO:teuthology.orchestra.run.smithi119.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:18:46.396 INFO:teuthology.orchestra.run.smithi119.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:18:46.396 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:46.396 INFO:teuthology.orchestra.run.smithi119.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:18:46.396 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:18:46.396 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:18:46.396 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:46.396 INFO:teuthology.orchestra.run.smithi119.stdout: # 1m isn't quite enough 2024-03-29T11:18:46.396 INFO:teuthology.orchestra.run.smithi119.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:18:46.396 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:46.396 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:18:46.396 INFO:teuthology.orchestra.run.smithi119.stdout: mon min osdmap epochs = 50 2024-03-29T11:18:46.397 INFO:teuthology.orchestra.run.smithi119.stdout: mon osdmap full prune interval = 2 2024-03-29T11:18:46.397 INFO:teuthology.orchestra.run.smithi119.stdout: mon osdmap full prune min = 15 2024-03-29T11:18:46.397 INFO:teuthology.orchestra.run.smithi119.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:18:46.397 INFO:teuthology.orchestra.run.smithi119.stdout: mon scrub interval = 300 2024-03-29T11:18:46.397 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on pool no app = False 2024-03-29T11:18:46.397 INFO:teuthology.orchestra.run.smithi119.stdout: paxos service trim min = 10 2024-03-29T11:18:46.397 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-03-29T11:18:46.397 INFO:teuthology.orchestra.run.smithi119.stdout:[client] 2024-03-29T11:18:46.397 INFO:teuthology.orchestra.run.smithi119.stdout: rgw cache enabled = true 2024-03-29T11:18:46.397 INFO:teuthology.orchestra.run.smithi119.stdout: rgw enable ops log = true 2024-03-29T11:18:46.397 INFO:teuthology.orchestra.run.smithi119.stdout: rgw enable usage log = true 2024-03-29T11:18:46.397 INFO:teuthology.orchestra.run.smithi119.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:18:46.397 INFO:teuthology.orchestra.run.smithi119.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:18:46.397 INFO:teuthology.orchestra.run.smithi119.stdout: debug monc = 20 2024-03-29T11:18:46.397 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2024-03-29T11:18:46.398 INFO:teuthology.orchestra.run.smithi119.stdout: debug objecter = 20 2024-03-29T11:18:46.398 INFO:teuthology.orchestra.run.smithi119.stdout: debug rados = 20 2024-03-29T11:18:46.398 INFO:teuthology.orchestra.run.smithi119.stdout:[mon.a] 2024-03-29T11:18:46.398 INFO:teuthology.orchestra.run.smithi119.stdout:[mon.c] 2024-03-29T11:18:46.398 INFO:teuthology.orchestra.run.smithi119.stdout:[mon.b] 2024-03-29T11:18:46.399 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T11:18:46.399 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T11:18:46.495 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T11:18:46.495 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-29T11:18:46.495 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T11:18:46.510 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-29T11:18:46.510 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T11:18:46.566 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi104.front.sepia.ceph.com 2024-03-29T11:18:46.566 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-29T11:18:46.566 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:18:46.567 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:18:46.645 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-29T11:18:46.645 DEBUG:teuthology.orchestra.run.smithi104:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:46.700 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi119.front.sepia.ceph.com 2024-03-29T11:18:46.700 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-29T11:18:46.701 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:18:46.701 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:18:46.741 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-29T11:18:46.742 DEBUG:teuthology.orchestra.run.smithi119:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:46.796 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T11:18:46.797 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T11:18:46.797 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:18:46.851 INFO:teuthology.orchestra.run.smithi104.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:18:46.853 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:18:46.918 INFO:teuthology.orchestra.run.smithi119.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:18:46.920 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T11:18:46.920 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T11:18:46.920 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T11:18:46.962 INFO:teuthology.orchestra.run.smithi104.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T11:18:46.977 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-29T11:18:47.019 INFO:teuthology.orchestra.run.smithi119.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-29T11:18:47.034 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T11:18:47.035 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi104.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi119.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:18:47.035 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T11:18:47.061 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:18:47.061 INFO:tasks.ceph:role: osd.0 2024-03-29T11:18:47.061 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi104.front.sepia.ceph.com 2024-03-29T11:18:47.061 DEBUG:teuthology.orchestra.run.smithi104:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:18:47.135 INFO:teuthology.orchestra.run.smithi104.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:18:47.136 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:18:47.136 INFO:teuthology.orchestra.run.smithi104.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:18:47.136 INFO:teuthology.orchestra.run.smithi104.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:18:47.137 INFO:teuthology.orchestra.run.smithi104.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:18:47.138 INFO:teuthology.orchestra.run.smithi104.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:18:47.138 INFO:teuthology.orchestra.run.smithi104.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:18:47.138 INFO:teuthology.orchestra.run.smithi104.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:18:47.138 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:18:47.138 INFO:teuthology.orchestra.run.smithi104.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:18:47.641 INFO:teuthology.orchestra.run.smithi104.stdout:Discarding blocks...Done. 2024-03-29T11:18:47.643 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi104.front.sepia.ceph.com -o noatime 2024-03-29T11:18:47.643 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T11:18:47.790 DEBUG:teuthology.orchestra.run.smithi104:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T11:18:47.827 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T11:18:47.896 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:18:47.896 INFO:tasks.ceph:role: osd.1 2024-03-29T11:18:47.896 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi104.front.sepia.ceph.com 2024-03-29T11:18:47.896 DEBUG:teuthology.orchestra.run.smithi104:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:18:47.971 INFO:teuthology.orchestra.run.smithi104.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:18:47.971 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:18:47.971 INFO:teuthology.orchestra.run.smithi104.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:18:47.971 INFO:teuthology.orchestra.run.smithi104.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:18:47.971 INFO:teuthology.orchestra.run.smithi104.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:18:47.971 INFO:teuthology.orchestra.run.smithi104.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:18:47.971 INFO:teuthology.orchestra.run.smithi104.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:18:47.971 INFO:teuthology.orchestra.run.smithi104.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:18:47.971 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:18:47.971 INFO:teuthology.orchestra.run.smithi104.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:18:48.419 INFO:teuthology.orchestra.run.smithi104.stdout:Discarding blocks...Done. 2024-03-29T11:18:48.420 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi104.front.sepia.ceph.com -o noatime 2024-03-29T11:18:48.421 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T11:18:48.514 DEBUG:teuthology.orchestra.run.smithi104:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T11:18:48.545 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T11:18:48.612 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:18:48.612 INFO:tasks.ceph:role: osd.2 2024-03-29T11:18:48.612 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi104.front.sepia.ceph.com 2024-03-29T11:18:48.613 DEBUG:teuthology.orchestra.run.smithi104:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:18:48.687 INFO:teuthology.orchestra.run.smithi104.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:18:48.687 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:18:48.687 INFO:teuthology.orchestra.run.smithi104.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:18:48.687 INFO:teuthology.orchestra.run.smithi104.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:18:48.687 INFO:teuthology.orchestra.run.smithi104.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:18:48.687 INFO:teuthology.orchestra.run.smithi104.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:18:48.687 INFO:teuthology.orchestra.run.smithi104.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:18:48.687 INFO:teuthology.orchestra.run.smithi104.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:18:48.688 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:18:48.688 INFO:teuthology.orchestra.run.smithi104.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:18:49.179 INFO:teuthology.orchestra.run.smithi104.stdout:Discarding blocks...Done. 2024-03-29T11:18:49.181 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi104.front.sepia.ceph.com -o noatime 2024-03-29T11:18:49.181 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T11:18:49.335 DEBUG:teuthology.orchestra.run.smithi104:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T11:18:49.367 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-29T11:18:49.434 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:18:49.434 INFO:tasks.ceph:role: osd.3 2024-03-29T11:18:49.434 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi104.front.sepia.ceph.com 2024-03-29T11:18:49.434 DEBUG:teuthology.orchestra.run.smithi104:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:18:49.508 INFO:teuthology.orchestra.run.smithi104.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:18:49.509 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:18:49.509 INFO:teuthology.orchestra.run.smithi104.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:18:49.509 INFO:teuthology.orchestra.run.smithi104.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:18:49.509 INFO:teuthology.orchestra.run.smithi104.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:18:49.509 INFO:teuthology.orchestra.run.smithi104.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:18:49.509 INFO:teuthology.orchestra.run.smithi104.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:18:49.509 INFO:teuthology.orchestra.run.smithi104.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:18:49.509 INFO:teuthology.orchestra.run.smithi104.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:18:49.509 INFO:teuthology.orchestra.run.smithi104.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:18:50.034 INFO:teuthology.orchestra.run.smithi104.stdout:Discarding blocks...Done. 2024-03-29T11:18:50.036 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi104.front.sepia.ceph.com -o noatime 2024-03-29T11:18:50.036 DEBUG:teuthology.orchestra.run.smithi104:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-29T11:18:50.169 DEBUG:teuthology.orchestra.run.smithi104:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-29T11:18:50.202 DEBUG:teuthology.orchestra.run.smithi104:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:50.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:18:50.326+0000 7f462b20f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-29T11:18:50.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:18:50.326+0000 7f462b20f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-29T11:18:50.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:18:50.326+0000 7f462b20f640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:18:50.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:18:50.326+0000 7f462b20f640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:18:50.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:18:50.326+0000 7f462b20f640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:18:50.304 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:18:50.327+0000 7f462b20f640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-29T11:18:53.156 DEBUG:teuthology.orchestra.run.smithi104:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T11:18:53.187 DEBUG:teuthology.orchestra.run.smithi104:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:53.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:18:53.301+0000 7f41f0dac640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-29T11:18:53.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:18:53.301+0000 7f41f0dac640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-29T11:18:53.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:18:53.301+0000 7f41f0dac640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:18:53.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:18:53.301+0000 7f41f0dac640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:18:53.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:18:53.301+0000 7f41f0dac640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:18:53.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:18:53.301+0000 7f41f0dac640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-29T11:18:56.116 DEBUG:teuthology.orchestra.run.smithi104:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T11:18:56.145 DEBUG:teuthology.orchestra.run.smithi104:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:56.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:18:56.254+0000 7fab904ae640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-29T11:18:56.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:18:56.255+0000 7fab904ae640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-29T11:18:56.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:18:56.255+0000 7fab904ae640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:18:56.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:18:56.255+0000 7fab904ae640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:18:56.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:18:56.255+0000 7fab904ae640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:18:56.232 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:18:56.255+0000 7fab904ae640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-29T11:18:59.077 DEBUG:teuthology.orchestra.run.smithi104:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T11:18:59.106 DEBUG:teuthology.orchestra.run.smithi104:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:59.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:18:59.218+0000 7f328920f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-29T11:18:59.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:18:59.218+0000 7f328920f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-29T11:18:59.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:18:59.218+0000 7f328920f640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:18:59.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:18:59.218+0000 7f328920f640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:18:59.195 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:18:59.218+0000 7f328920f640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:18:59.196 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:18:59.218+0000 7f328920f640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-29T11:19:02.044 DEBUG:teuthology.orchestra.run.smithi104:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-29T11:19:02.073 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-29T11:19:02.105 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:02.105 INFO:tasks.ceph:role: osd.4 2024-03-29T11:19:02.105 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi119.front.sepia.ceph.com 2024-03-29T11:19:02.105 DEBUG:teuthology.orchestra.run.smithi119:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:19:02.183 INFO:teuthology.orchestra.run.smithi119.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:19:02.183 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:19:02.183 INFO:teuthology.orchestra.run.smithi119.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:19:02.183 INFO:teuthology.orchestra.run.smithi119.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:19:02.183 INFO:teuthology.orchestra.run.smithi119.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:19:02.183 INFO:teuthology.orchestra.run.smithi119.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:19:02.184 INFO:teuthology.orchestra.run.smithi119.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:19:02.184 INFO:teuthology.orchestra.run.smithi119.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:19:02.184 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:19:02.184 INFO:teuthology.orchestra.run.smithi119.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:19:02.668 INFO:teuthology.orchestra.run.smithi119.stdout:Discarding blocks...Done. 2024-03-29T11:19:02.670 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi119.front.sepia.ceph.com -o noatime 2024-03-29T11:19:02.670 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-29T11:19:02.780 DEBUG:teuthology.orchestra.run.smithi119:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-29T11:19:02.813 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-29T11:19:02.881 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:02.881 INFO:tasks.ceph:role: osd.5 2024-03-29T11:19:02.881 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi119.front.sepia.ceph.com 2024-03-29T11:19:02.881 DEBUG:teuthology.orchestra.run.smithi119:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:19:02.955 INFO:teuthology.orchestra.run.smithi119.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:19:02.955 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:19:02.955 INFO:teuthology.orchestra.run.smithi119.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:19:02.955 INFO:teuthology.orchestra.run.smithi119.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:19:02.955 INFO:teuthology.orchestra.run.smithi119.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:19:02.955 INFO:teuthology.orchestra.run.smithi119.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:19:02.955 INFO:teuthology.orchestra.run.smithi119.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:19:02.956 INFO:teuthology.orchestra.run.smithi119.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:19:02.956 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:19:02.956 INFO:teuthology.orchestra.run.smithi119.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:19:03.404 INFO:teuthology.orchestra.run.smithi119.stdout:Discarding blocks...Done. 2024-03-29T11:19:03.406 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi119.front.sepia.ceph.com -o noatime 2024-03-29T11:19:03.406 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-29T11:19:03.479 DEBUG:teuthology.orchestra.run.smithi119:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-29T11:19:03.551 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-29T11:19:03.618 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:03.619 INFO:tasks.ceph:role: osd.6 2024-03-29T11:19:03.619 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi119.front.sepia.ceph.com 2024-03-29T11:19:03.619 DEBUG:teuthology.orchestra.run.smithi119:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:19:03.692 INFO:teuthology.orchestra.run.smithi119.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:19:03.693 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:19:03.693 INFO:teuthology.orchestra.run.smithi119.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:19:03.693 INFO:teuthology.orchestra.run.smithi119.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:19:03.693 INFO:teuthology.orchestra.run.smithi119.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:19:03.693 INFO:teuthology.orchestra.run.smithi119.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:19:03.693 INFO:teuthology.orchestra.run.smithi119.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:19:03.693 INFO:teuthology.orchestra.run.smithi119.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:19:03.693 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:19:03.693 INFO:teuthology.orchestra.run.smithi119.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:19:04.180 INFO:teuthology.orchestra.run.smithi119.stdout:Discarding blocks...Done. 2024-03-29T11:19:04.182 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi119.front.sepia.ceph.com -o noatime 2024-03-29T11:19:04.182 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-29T11:19:04.271 DEBUG:teuthology.orchestra.run.smithi119:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-29T11:19:04.302 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-29T11:19:04.370 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:04.370 INFO:tasks.ceph:role: osd.7 2024-03-29T11:19:04.370 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi119.front.sepia.ceph.com 2024-03-29T11:19:04.370 DEBUG:teuthology.orchestra.run.smithi119:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:19:04.444 INFO:teuthology.orchestra.run.smithi119.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:19:04.444 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:19:04.444 INFO:teuthology.orchestra.run.smithi119.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:19:04.444 INFO:teuthology.orchestra.run.smithi119.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:19:04.444 INFO:teuthology.orchestra.run.smithi119.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:19:04.444 INFO:teuthology.orchestra.run.smithi119.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:19:04.444 INFO:teuthology.orchestra.run.smithi119.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:19:04.444 INFO:teuthology.orchestra.run.smithi119.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:19:04.445 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:19:04.445 INFO:teuthology.orchestra.run.smithi119.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:19:04.964 INFO:teuthology.orchestra.run.smithi119.stdout:Discarding blocks...Done. 2024-03-29T11:19:04.966 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi119.front.sepia.ceph.com -o noatime 2024-03-29T11:19:04.966 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-29T11:19:05.116 DEBUG:teuthology.orchestra.run.smithi119:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-29T11:19:05.152 DEBUG:teuthology.orchestra.run.smithi119:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:05.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:05.273+0000 7f0580e0e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-29T11:19:05.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:05.277+0000 7f0580e0e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-29T11:19:05.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:05.277+0000 7f0580e0e640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:19:05.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:05.277+0000 7f0580e0e640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:19:05.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:05.277+0000 7f0580e0e640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:19:05.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:05.286+0000 7f0580e0e640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-29T11:19:08.115 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-29T11:19:08.146 DEBUG:teuthology.orchestra.run.smithi119:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:08.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:08.260+0000 7fe10a20d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-29T11:19:08.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:08.260+0000 7fe10a20d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-29T11:19:08.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:08.260+0000 7fe10a20d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:19:08.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:08.260+0000 7fe10a20d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:19:08.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:08.260+0000 7fe10a20d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:19:08.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:08.260+0000 7fe10a20d640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-29T11:19:11.090 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-29T11:19:11.121 DEBUG:teuthology.orchestra.run.smithi119:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:11.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:11.237+0000 7f6564b1f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-29T11:19:11.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:11.237+0000 7f6564b1f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-29T11:19:11.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:11.237+0000 7f6564b1f640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:19:11.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:11.237+0000 7f6564b1f640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:19:11.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:11.237+0000 7f6564b1f640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:19:11.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:11.238+0000 7f6564b1f640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-29T11:19:14.065 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-29T11:19:14.095 DEBUG:teuthology.orchestra.run.smithi119:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:14.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:14.206+0000 7f03cb610640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-29T11:19:14.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:14.206+0000 7f03cb610640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-29T11:19:14.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:14.207+0000 7f03cb610640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:19:14.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:14.207+0000 7f03cb610640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:19:14.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:14.207+0000 7f03cb610640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:19:14.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:14.207+0000 7f03cb610640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-29T11:19:17.034 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-29T11:19:17.065 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T11:19:17.065 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-29T11:19:17.065 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-29T11:19:17.097 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-29T11:19:17.097 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T11:19:17.168 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-29T11:19:17.169 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T11:19:17.238 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-29T11:19:17.238 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T11:19:17.307 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-29T11:19:17.308 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-29T11:19:17.377 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-29T11:19:17.377 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T11:19:17.408 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-29T11:19:17.409 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-29T11:19:17.480 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-29T11:19:17.480 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-29T11:19:17.552 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-29T11:19:17.552 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-29T11:19:17.622 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-29T11:19:17.622 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-29T11:19:17.692 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-29T11:19:17.692 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T11:19:17.709 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-29T11:19:17.709 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-29T11:19:17.748 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T11:19:17.748 DEBUG:teuthology.orchestra.run.smithi104:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:19:17.752 DEBUG:teuthology.orchestra.run.smithi119:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:19:17.791 INFO:teuthology.orchestra.run.smithi104.stdout:[mgr.y] 2024-03-29T11:19:17.791 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCWowZmPvsvNBAAhb2Q9raG+6WGdCPPf7QOng== 2024-03-29T11:19:17.791 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.0] 2024-03-29T11:19:17.791 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCaowZmisOFExAAZWevGjWt1cplL5/qgsd84A== 2024-03-29T11:19:17.791 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.1] 2024-03-29T11:19:17.791 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCdowZm/coAEhAAGfldYYUqi63hQzOK+f83GA== 2024-03-29T11:19:17.791 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.2] 2024-03-29T11:19:17.791 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCgowZmZ0RADxAAl1aIs7FWSoNJm/tqrgQwGg== 2024-03-29T11:19:17.791 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.3] 2024-03-29T11:19:17.791 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCjowZmpqYNDRAABZcDy6waEaOnNjpER7UFzg== 2024-03-29T11:19:17.791 INFO:teuthology.orchestra.run.smithi104.stdout:[mgr.x] 2024-03-29T11:19:17.792 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCWowZmjt0pOBAAJ+O8biCyckPLXumFz4NgxA== 2024-03-29T11:19:17.792 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.4] 2024-03-29T11:19:17.792 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCpowZmpj1kEBAArtFFYXnGmbo8W+UlamFLlg== 2024-03-29T11:19:17.792 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.5] 2024-03-29T11:19:17.792 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCsowZmTauSDxAAaCONrAHLrm5mcwKoM08mRA== 2024-03-29T11:19:17.792 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.6] 2024-03-29T11:19:17.792 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCvowZmJVY6DhAAJiKFTkWIgzRATdS4p7vjsw== 2024-03-29T11:19:17.792 INFO:teuthology.orchestra.run.smithi104.stdout:[osd.7] 2024-03-29T11:19:17.792 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCyowZmebpVDBAAOe0s61UJ7pE3IfHVw2pN0w== 2024-03-29T11:19:17.792 INFO:teuthology.orchestra.run.smithi104.stdout:[client.0] 2024-03-29T11:19:17.792 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCWowZmfTzTOhAA0i8++2apnIyVAE3+cmRMzw== 2024-03-29T11:19:17.792 INFO:teuthology.orchestra.run.smithi104.stdout:[client.1] 2024-03-29T11:19:17.792 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCXowZmALKgAhAA2PCwrr3dBxKquaYjefkgOA== 2024-03-29T11:19:17.814 INFO:teuthology.orchestra.run.smithi119.stdout:[mgr.y] 2024-03-29T11:19:17.814 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQCWowZmPvsvNBAAhb2Q9raG+6WGdCPPf7QOng== 2024-03-29T11:19:17.815 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.0] 2024-03-29T11:19:17.815 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQCaowZmisOFExAAZWevGjWt1cplL5/qgsd84A== 2024-03-29T11:19:17.815 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.1] 2024-03-29T11:19:17.815 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQCdowZm/coAEhAAGfldYYUqi63hQzOK+f83GA== 2024-03-29T11:19:17.815 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.2] 2024-03-29T11:19:17.815 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQCgowZmZ0RADxAAl1aIs7FWSoNJm/tqrgQwGg== 2024-03-29T11:19:17.815 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.3] 2024-03-29T11:19:17.815 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQCjowZmpqYNDRAABZcDy6waEaOnNjpER7UFzg== 2024-03-29T11:19:17.815 INFO:teuthology.orchestra.run.smithi119.stdout:[mgr.x] 2024-03-29T11:19:17.815 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQCWowZmjt0pOBAAJ+O8biCyckPLXumFz4NgxA== 2024-03-29T11:19:17.815 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.4] 2024-03-29T11:19:17.815 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQCpowZmpj1kEBAArtFFYXnGmbo8W+UlamFLlg== 2024-03-29T11:19:17.815 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.5] 2024-03-29T11:19:17.815 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQCsowZmTauSDxAAaCONrAHLrm5mcwKoM08mRA== 2024-03-29T11:19:17.815 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.6] 2024-03-29T11:19:17.815 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQCvowZmJVY6DhAAJiKFTkWIgzRATdS4p7vjsw== 2024-03-29T11:19:17.816 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.7] 2024-03-29T11:19:17.816 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQCyowZmebpVDBAAOe0s61UJ7pE3IfHVw2pN0w== 2024-03-29T11:19:17.816 INFO:teuthology.orchestra.run.smithi119.stdout:[client.0] 2024-03-29T11:19:17.816 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQCWowZmfTzTOhAA0i8++2apnIyVAE3+cmRMzw== 2024-03-29T11:19:17.816 INFO:teuthology.orchestra.run.smithi119.stdout:[client.1] 2024-03-29T11:19:17.816 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQCXowZmALKgAhAA2PCwrr3dBxKquaYjefkgOA== 2024-03-29T11:19:17.817 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:19:17.836 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:19:17.910 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:17.912 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:17.964 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:17.966 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:18.018 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:18.020 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:18.071 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:18.074 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:18.127 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:19:18.131 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:19:18.182 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:18.185 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:18.236 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:18.240 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:18.291 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:18.294 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:18.345 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:18.348 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:18.399 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:19:18.401 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:19:18.455 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:19:18.458 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:19:18.512 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T11:19:18.513 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T11:19:18.540 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:19:18.637 DEBUG:teuthology.orchestra.run.smithi104:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T11:19:18.664 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-29T11:19:18.731 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:19:18.828 DEBUG:teuthology.orchestra.run.smithi104:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-29T11:19:18.855 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-29T11:19:18.882 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:19:18.979 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-29T11:19:19.006 DEBUG:teuthology.orchestra.run.smithi104:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:19.008 DEBUG:teuthology.orchestra.run.smithi119:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:19.060 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T11:19:19.061 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T11:19:19.061 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T11:19:19.063 INFO:tasks.ceph.mon.a:Started 2024-03-29T11:19:19.063 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-29T11:19:19.063 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-29T11:19:19.065 INFO:tasks.ceph.mon.c:Started 2024-03-29T11:19:19.065 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-29T11:19:19.065 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-29T11:19:19.103 INFO:tasks.ceph.mon.b:Started 2024-03-29T11:19:19.103 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T11:19:19.103 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-29T11:19:19.103 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-29T11:19:19.105 INFO:tasks.ceph.mgr.y:Started 2024-03-29T11:19:19.106 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T11:19:19.106 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T11:19:19.107 INFO:tasks.ceph.mgr.x:Started 2024-03-29T11:19:19.108 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T11:19:19.108 DEBUG:teuthology.orchestra.run.smithi104:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T11:19:19.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:19.232+0000 7f8cbd11c640 10 monclient: get_monmap_and_config 2024-03-29T11:19:19.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:19.232+0000 7f8cbd11c640 10 monclient: build_initial_monmap 2024-03-29T11:19:19.209 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:19.232+0000 7f8cbd11c640 10 monclient: monmap: 2024-03-29T11:19:19.209 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-03-29T11:19:19.209 INFO:teuthology.orchestra.run.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:19.209 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-03-29T11:19:19.233024+0000 2024-03-29T11:19:19.209 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-03-29T11:19:19.233024+0000 2024-03-29T11:19:19.209 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:19.209 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-03-29T11:19:19.209 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:19:19.209 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:19:19.210 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:19:19.210 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-03-29T11:19:19.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:19.232+0000 7f8cbd11c640 1 Processor -- start 2024-03-29T11:19:19.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:19.232+0000 7f8cbd11c640 1 -- start start 2024-03-29T11:19:19.210 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:19.232+0000 7f8cbd11c640 10 monclient: init 2024-03-29T11:19:19.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:19.233+0000 7f8cbd11c640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:19.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:19.233+0000 7f8cbd11c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:19.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:19.233+0000 7f8cbd11c640 10 monclient(hunting): picked mon.noname-a con 0x7f8cb80b8240 addr v1:172.21.15.104:6789/0 2024-03-29T11:19:19.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:19.233+0000 7f8cbd11c640 10 monclient(hunting): picked mon.noname-c con 0x7f8cb80b1f60 addr v1:172.21.15.119:6789/0 2024-03-29T11:19:19.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:19.233+0000 7f8cbd11c640 10 monclient(hunting): picked mon.noname-b con 0x7f8cb809c8c0 addr v1:172.21.15.104:6790/0 2024-03-29T11:19:19.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:19.233+0000 7f8cbd11c640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb809bd80 con 0x7f8cb80b8240 2024-03-29T11:19:19.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:19.233+0000 7f8cbd11c640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb809b960 con 0x7f8cb80b1f60 2024-03-29T11:19:19.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:19.233+0000 7f8cbd11c640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb80b2ec0 con 0x7f8cb809c8c0 2024-03-29T11:19:19.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:19.233+0000 7f8cbd11c640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:19.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:19.233+0000 7f8cbd11c640 10 monclient(hunting): authenticate will time out at 940.484463s 2024-03-29T11:19:19.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:19.233+0000 7f8cb759e640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f8cb809c8c0 0x7f8cb814d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:37848/0 (socket says 172.21.15.104:37848) 2024-03-29T11:19:19.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:19.233+0000 7f8cb6d9d640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f8cb80b8240 0x7f8cb813f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:38246/0 (socket says 172.21.15.104:38246) 2024-03-29T11:19:19.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:19.233+0000 7f8cb759e640 1 -- 172.21.15.104:0/1659348868 learned_addr learned my addr 172.21.15.104:0/1659348868 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:19:22.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:22.234+0000 7f8cb4d99640 10 monclient(hunting): tick 2024-03-29T11:19:22.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:22.234+0000 7f8cb4d99640 10 monclient(hunting): _check_auth_tickets 2024-03-29T11:19:22.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:22.234+0000 7f8cb4d99640 1 monclient(hunting): continuing hunt 2024-03-29T11:19:22.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:22.234+0000 7f8cb4d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-29T11:19:22.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:22.234+0000 7f8cb4d99640 1 -- 172.21.15.104:0/1659348868 >> v1:172.21.15.104:6790/0 conn(0x7f8cb809c8c0 legacy=0x7f8cb814d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:22.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:22.234+0000 7f8cb4d99640 1 -- 172.21.15.104:0/1659348868 >> v1:172.21.15.119:6789/0 conn(0x7f8cb80b1f60 legacy=0x7f8cb809c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:22.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:22.234+0000 7f8cb4d99640 1 -- 172.21.15.104:0/1659348868 >> v1:172.21.15.104:6789/0 conn(0x7f8cb80b8240 legacy=0x7f8cb813f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:22.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:22.234+0000 7f8cb4d99640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:22.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:22.234+0000 7f8cb4d99640 10 monclient(hunting): picked mon.noname-a con 0x7f8ca40036e0 addr v1:172.21.15.104:6789/0 2024-03-29T11:19:22.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:22.234+0000 7f8cb4d99640 10 monclient(hunting): picked mon.noname-b con 0x7f8cb809c8c0 addr v1:172.21.15.104:6790/0 2024-03-29T11:19:22.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:22.234+0000 7f8cb4d99640 10 monclient(hunting): picked mon.noname-c con 0x7f8ca400aab0 addr v1:172.21.15.119:6789/0 2024-03-29T11:19:22.211 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:22.234+0000 7f8cb4d99640 1 -- 172.21.15.104:0/1659348868 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb813fb60 con 0x7f8ca40036e0 2024-03-29T11:19:22.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:22.234+0000 7f8cb4d99640 1 -- 172.21.15.104:0/1659348868 --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ca400f760 con 0x7f8ca400aab0 2024-03-29T11:19:22.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:22.234+0000 7f8cb4d99640 1 -- 172.21.15.104:0/1659348868 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ca40109c0 con 0x7f8cb809c8c0 2024-03-29T11:19:22.212 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:22.234+0000 7f8cb4d99640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:24.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.407+0000 7f8cb5d9b640 1 -- 172.21.15.104:0/1659348868 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 174982782 0 0) 0x7f8cb813fb60 con 0x7f8ca40036e0 2024-03-29T11:19:24.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.407+0000 7f8cb5d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:19:24.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.407+0000 7f8cb5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:24.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.407+0000 7f8cb5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:24.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.407+0000 7f8cb5d9b640 10 monclient(hunting): my global_id is 4101 2024-03-29T11:19:24.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.407+0000 7f8cb5d9b640 1 -- 172.21.15.104:0/1659348868 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ca0003520 con 0x7f8ca40036e0 2024-03-29T11:19:24.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.407+0000 7f8cb5d9b640 1 -- 172.21.15.104:0/1659348868 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3460249821 0 0) 0x7f8ca40109c0 con 0x7f8cb809c8c0 2024-03-29T11:19:24.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.407+0000 7f8cb5d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:19:24.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.407+0000 7f8cb5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:24.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.407+0000 7f8cb5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:24.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.407+0000 7f8cb5d9b640 10 monclient(hunting): my global_id is 4106 2024-03-29T11:19:24.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.407+0000 7f8cb5d9b640 1 -- 172.21.15.104:0/1659348868 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cb813fb60 con 0x7f8cb809c8c0 2024-03-29T11:19:24.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.407+0000 7f8cb5d9b640 1 -- 172.21.15.104:0/1659348868 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4143561616 0 0) 0x7f8cb813fb60 con 0x7f8cb809c8c0 2024-03-29T11:19:24.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.407+0000 7f8cb5d9b640 1 -- 172.21.15.104:0/1659348868 >> v1:172.21.15.119:6789/0 conn(0x7f8ca400aab0 legacy=0x7f8ca400cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:24.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.407+0000 7f8cb5d9b640 1 -- 172.21.15.104:0/1659348868 >> v1:172.21.15.104:6789/0 conn(0x7f8ca40036e0 legacy=0x7f8ca4005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:24.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.407+0000 7f8cb5d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:19:24.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.407+0000 7f8cb5d9b640 1 monclient: found mon.noname-b 2024-03-29T11:19:24.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.407+0000 7f8cb5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:24.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.407+0000 7f8cb5d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.104:6790/0 2024-03-29T11:19:24.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.407+0000 7f8cb5d9b640 1 -- 172.21.15.104:0/1659348868 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ca4011c20 con 0x7f8cb809c8c0 2024-03-29T11:19:24.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.407+0000 7f8cb5d9b640 10 monclient: _finish_auth 0 2024-03-29T11:19:24.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.407+0000 7f8cb5d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:19:24.386 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.407+0000 7f8cb5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:24.387 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.407+0000 7f8cbd11c640 5 monclient: authenticate success, global_id 4106 2024-03-29T11:19:24.387 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.407+0000 7f8cbd11c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:24.387 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.407+0000 7f8cb5d9b640 1 -- 172.21.15.104:0/1659348868 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f8cac001610 con 0x7f8cb809c8c0 2024-03-29T11:19:24.387 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.407+0000 7f8cb5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:24.387 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.408+0000 7f8cb5d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:24.387 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.408+0000 7f8cb5d9b640 10 monclient: dump: 2024-03-29T11:19:24.387 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-03-29T11:19:24.387 INFO:teuthology.orchestra.run.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:24.387 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:24.387 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:24.387 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:24.387 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-03-29T11:19:24.387 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:24.387 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:24.387 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:24.388 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-03-29T11:19:24.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.408+0000 7f8cbd11c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:24.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.408+0000 7f8cb5d9b640 1 -- 172.21.15.104:0/1659348868 <== mon.2 v1:172.21.15.104:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8cac0017e0 con 0x7f8cb809c8c0 2024-03-29T11:19:24.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.408+0000 7f8cb5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:24.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.408+0000 7f8cb5d9b640 1 -- 172.21.15.104:0/1659348868 <== mon.2 v1:172.21.15.104:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f8cac001b70 con 0x7f8cb809c8c0 2024-03-29T11:19:24.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.408+0000 7f8cb5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:24.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.408+0000 7f8cb5d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:24.388 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.408+0000 7f8cb5d9b640 10 monclient: dump: 2024-03-29T11:19:24.388 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-03-29T11:19:24.388 INFO:teuthology.orchestra.run.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:24.388 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:24.388 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:24.388 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:24.388 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-03-29T11:19:24.388 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:24.389 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:24.389 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:24.389 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-03-29T11:19:24.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.408+0000 7f8cbd11c640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:24.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.408+0000 7f8cbd11c640 4 set_mon_vals no callback set 2024-03-29T11:19:24.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.409+0000 7f8cbd11c640 10 monclient: shutdown 2024-03-29T11:19:24.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.409+0000 7f8cbd11c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:24.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.409+0000 7f8cbd11c640 1 -- 172.21.15.104:0/1659348868 >> v1:172.21.15.104:6790/0 conn(0x7f8cb809c8c0 legacy=0x7f8ca4009340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:24.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.409+0000 7f8cbd11c640 1 -- 172.21.15.104:0/1659348868 shutdown_connections 2024-03-29T11:19:24.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.409+0000 7f8cbd11c640 1 -- 172.21.15.104:0/1659348868 >> 172.21.15.104:0/1659348868 conn(0x7f8cb8066b00 msgr2=0x7f8cb8066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:24.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.409+0000 7f8cbd11c640 1 -- 172.21.15.104:0/1659348868 shutdown_connections 2024-03-29T11:19:24.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.409+0000 7f8cbd11c640 1 -- 172.21.15.104:0/1659348868 wait complete. 2024-03-29T11:19:24.395 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.410+0000 7f8cbd11c640 10 monclient: build_initial_monmap 2024-03-29T11:19:24.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.410+0000 7f8cbd11c640 10 monclient: monmap: 2024-03-29T11:19:24.396 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-03-29T11:19:24.396 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:24.396 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-03-29T11:19:24.396 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-03-29T11:19:24.396 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:24.396 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-03-29T11:19:24.396 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:19:24.396 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:19:24.396 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:19:24.396 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-03-29T11:19:24.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.410+0000 7f8cbd11c640 1 librados: starting msgr at 2024-03-29T11:19:24.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.410+0000 7f8cbd11c640 1 librados: starting objecter 2024-03-29T11:19:24.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.411+0000 7f8cbd11c640 1 Processor -- start 2024-03-29T11:19:24.396 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.411+0000 7f8cbd11c640 1 -- start start 2024-03-29T11:19:24.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.411+0000 7f8cbd11c640 1 librados: setting wanted keys 2024-03-29T11:19:24.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.411+0000 7f8cbd11c640 1 librados: calling monclient init 2024-03-29T11:19:24.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.411+0000 7f8cbd11c640 10 monclient: init 2024-03-29T11:19:24.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.412+0000 7f8cbd11c640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:24.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.412+0000 7f8cbd11c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:24.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.412+0000 7f8cbd11c640 10 monclient(hunting): picked mon.noname-b con 0x7f8ca40036e0 addr v1:172.21.15.119:6789/0 2024-03-29T11:19:24.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.412+0000 7f8cbd11c640 10 monclient(hunting): picked mon.noname-a con 0x7f8ca400aab0 addr v1:172.21.15.104:6789/0 2024-03-29T11:19:24.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.412+0000 7f8cbd11c640 10 monclient(hunting): picked mon.noname-c con 0x7f8cb809c8c0 addr v1:172.21.15.104:6790/0 2024-03-29T11:19:24.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.412+0000 7f8cbd11c640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb81d6fe0 con 0x7f8ca400aab0 2024-03-29T11:19:24.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.412+0000 7f8cbd11c640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb80ad140 con 0x7f8ca40036e0 2024-03-29T11:19:24.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.412+0000 7f8cbd11c640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb80b1cb0 con 0x7f8cb809c8c0 2024-03-29T11:19:24.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.412+0000 7f8cbd11c640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:24.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.412+0000 7f8cbd11c640 10 monclient(hunting): authenticate will time out at 945.663289s 2024-03-29T11:19:24.397 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.412+0000 7f8cb659c640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f8ca400aab0 0x7f8cb8138fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:42340/0 (socket says 172.21.15.104:42340) 2024-03-29T11:19:24.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.412+0000 7f8cb659c640 1 -- 172.21.15.104:0/3661510795 learned_addr learned my addr 172.21.15.104:0/3661510795 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:19:24.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.412+0000 7f8c9f7fe640 10 client.?.objecter ms_handle_connect 0x7f8cb809c8c0 2024-03-29T11:19:24.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.412+0000 7f8c9f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:24.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.412+0000 7f8c9f7fe640 10 client.?.objecter ms_handle_connect 0x7f8ca400aab0 2024-03-29T11:19:24.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.412+0000 7f8c9f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:24.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.412+0000 7f8c9f7fe640 10 client.?.objecter ms_handle_connect 0x7f8ca40036e0 2024-03-29T11:19:24.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.412+0000 7f8c9f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:24.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.413+0000 7f8c9f7fe640 1 -- 172.21.15.104:0/3661510795 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1674047671 0 0) 0x7f8cb80ad140 con 0x7f8ca40036e0 2024-03-29T11:19:24.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.413+0000 7f8c9f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:24.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.413+0000 7f8c9f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:24.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.413+0000 7f8c9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:24.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.413+0000 7f8c9f7fe640 10 monclient(hunting): my global_id is 4099 2024-03-29T11:19:24.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.413+0000 7f8c9f7fe640 1 -- 172.21.15.104:0/3661510795 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ca40100b0 con 0x7f8ca40036e0 2024-03-29T11:19:24.398 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.413+0000 7f8c9f7fe640 1 -- 172.21.15.104:0/3661510795 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2697901098 0 0) 0x7f8cb80b1cb0 con 0x7f8cb809c8c0 2024-03-29T11:19:24.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.413+0000 7f8c9f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:24.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.413+0000 7f8c9f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:24.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.413+0000 7f8c9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:24.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.413+0000 7f8c9f7fe640 10 monclient(hunting): my global_id is 4109 2024-03-29T11:19:24.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.413+0000 7f8c9f7fe640 1 -- 172.21.15.104:0/3661510795 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cb80ad140 con 0x7f8cb809c8c0 2024-03-29T11:19:24.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.413+0000 7f8cb759e640 1 -- 172.21.15.104:0/3661510795 >> v1:172.21.15.104:6790/0 conn(0x7f8cb809c8c0 legacy=0x7f8cb81e2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-29T11:19:24.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.413+0000 7f8cb759e640 1 -- 172.21.15.104:0/3661510795 >> v1:172.21.15.104:6790/0 conn(0x7f8cb809c8c0 legacy=0x7f8cb81e2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:19:24.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.413+0000 7f8cb759e640 1 --1- 172.21.15.104:0/3661510795 >> v1:172.21.15.104:6790/0 conn(0x7f8cb809c8c0 0x7f8cb81e2900 :-1 s=OPENED pgs=17 cs=1 l=1).handle_message read tag failed 2024-03-29T11:19:24.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.413+0000 7f8cb759e640 1 --1- 172.21.15.104:0/3661510795 >> v1:172.21.15.104:6790/0 conn(0x7f8cb809c8c0 0x7f8cb81e2900 :-1 s=OPENED pgs=17 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:19:24.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.413+0000 7f8c9f7fe640 1 -- 172.21.15.104:0/3661510795 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2657363678 0 0) 0x7f8ca40100b0 con 0x7f8ca40036e0 2024-03-29T11:19:24.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.413+0000 7f8c9f7fe640 1 -- 172.21.15.104:0/3661510795 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8cb80b1cb0 con 0x7f8ca40036e0 2024-03-29T11:19:24.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.413+0000 7f8c9f7fe640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.104:6790/0 2024-03-29T11:19:24.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.413+0000 7f8c9f7fe640 1 -- 172.21.15.104:0/3661510795 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f8ca80018f0 con 0x7f8ca40036e0 2024-03-29T11:19:24.399 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.413+0000 7f8c9f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:24.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.413+0000 7f8c9f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:24.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.413+0000 7f8c9f7fe640 10 monclient(hunting): dump: 2024-03-29T11:19:24.400 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-03-29T11:19:24.400 INFO:teuthology.orchestra.run.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:24.400 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:24.400 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:24.400 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:24.400 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-03-29T11:19:24.400 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:24.400 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:24.400 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:24.400 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-03-29T11:19:24.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.413+0000 7f8c9f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:24.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.414+0000 7f8c9f7fe640 1 -- 172.21.15.104:0/3661510795 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2033122791 0 0) 0x7f8cb81d6fe0 con 0x7f8ca400aab0 2024-03-29T11:19:24.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.414+0000 7f8c9f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:24.400 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.414+0000 7f8c9f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:24.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.414+0000 7f8c9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:24.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.414+0000 7f8c9f7fe640 10 monclient(hunting): my global_id is 4113 2024-03-29T11:19:24.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.414+0000 7f8c9f7fe640 1 -- 172.21.15.104:0/3661510795 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ca4002e80 con 0x7f8ca400aab0 2024-03-29T11:19:24.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.414+0000 7f8c9f7fe640 1 -- 172.21.15.104:0/3661510795 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3787030486 0 0) 0x7f8ca4002e80 con 0x7f8ca400aab0 2024-03-29T11:19:24.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.414+0000 7f8c9f7fe640 1 -- 172.21.15.104:0/3661510795 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8cb81d6fe0 con 0x7f8ca400aab0 2024-03-29T11:19:24.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.414+0000 7f8c9f7fe640 1 -- 172.21.15.104:0/3661510795 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f8cac001af0 con 0x7f8ca400aab0 2024-03-29T11:19:24.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.414+0000 7f8c9f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:24.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.414+0000 7f8c9f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:24.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.414+0000 7f8c9f7fe640 10 monclient(hunting): dump: 2024-03-29T11:19:24.401 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-03-29T11:19:24.401 INFO:teuthology.orchestra.run.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:24.401 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:24.401 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:24.401 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:24.402 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-03-29T11:19:24.402 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:24.402 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:24.402 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:24.402 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-03-29T11:19:24.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.414+0000 7f8c9f7fe640 1 -- 172.21.15.104:0/3661510795 <== mon.0 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 635930391 0 0) 0x7f8cb81d6fe0 con 0x7f8ca400aab0 2024-03-29T11:19:24.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.414+0000 7f8c9f7fe640 1 -- 172.21.15.104:0/3661510795 >> v1:172.21.15.104:6790/0 conn(0x7f8cb809c8c0 legacy=0x7f8cb81e2900 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:19:24.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.414+0000 7f8c9f7fe640 1 -- 172.21.15.104:0/3661510795 >> v1:172.21.15.119:6789/0 conn(0x7f8ca40036e0 legacy=0x7f8cb81d7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:24.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.414+0000 7f8c9f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:24.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.414+0000 7f8c9f7fe640 1 monclient: found mon.a 2024-03-29T11:19:24.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.414+0000 7f8c9f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:24.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.414+0000 7f8c9f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:19:24.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.414+0000 7f8c9f7fe640 1 -- 172.21.15.104:0/3661510795 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb80b0cc0 con 0x7f8ca400aab0 2024-03-29T11:19:24.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.414+0000 7f8c9f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:19:24.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.414+0000 7f8c9f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:24.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.414+0000 7f8c9f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:24.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.414+0000 7f8cbd11c640 5 monclient: authenticate success, global_id 4113 2024-03-29T11:19:24.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.414+0000 7f8cbd11c640 10 monclient: _renew_subs 2024-03-29T11:19:24.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.414+0000 7f8cbd11c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:19:24.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.414+0000 7f8cbd11c640 1 -- 172.21.15.104:0/3661510795 --> v1:172.21.15.104:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cb81d7b80 con 0x7f8ca400aab0 2024-03-29T11:19:24.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.414+0000 7f8cbd11c640 10 client.4113.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:24.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.414+0000 7f8cbd11c640 10 monclient: _renew_subs 2024-03-29T11:19:24.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.414+0000 7f8cbd11c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:19:24.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.414+0000 7f8cbd11c640 1 -- 172.21.15.104:0/3661510795 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cb80af8c0 con 0x7f8ca400aab0 2024-03-29T11:19:24.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.414+0000 7f8cbd11c640 1 librados: init done 2024-03-29T11:19:24.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.414+0000 7f8c9f7fe640 1 -- 172.21.15.104:0/3661510795 <== mon.0 v1:172.21.15.104:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8cac001cb0 con 0x7f8ca400aab0 2024-03-29T11:19:24.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.414+0000 7f8c9f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:24.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.414+0000 7f8c9f7fe640 1 -- 172.21.15.104:0/3661510795 <== mon.0 v1:172.21.15.104:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f8cac003a40 con 0x7f8ca400aab0 2024-03-29T11:19:24.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.414+0000 7f8c9f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:24.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.414+0000 7f8c9f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:24.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.414+0000 7f8cb559a640 4 set_mon_vals no callback set 2024-03-29T11:19:24.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.414+0000 7f8c9f7fe640 10 monclient: dump: 2024-03-29T11:19:24.404 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-03-29T11:19:24.404 INFO:teuthology.orchestra.run.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:24.404 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:24.404 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:24.404 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:24.404 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-03-29T11:19:24.404 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:24.404 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:24.404 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:24.404 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-03-29T11:19:24.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.414+0000 7f8c9f7fe640 1 -- 172.21.15.104:0/3661510795 <== mon.0 v1:172.21.15.104:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f8cac003ff0 con 0x7f8ca400aab0 2024-03-29T11:19:24.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.414+0000 7f8c9f7fe640 1 -- 172.21.15.104:0/3661510795 <== mon.0 v1:172.21.15.104:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 4150179697 0 0) 0x7f8cac0063d0 con 0x7f8ca400aab0 2024-03-29T11:19:24.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.414+0000 7f8c9f7fe640 10 client.4113.objecter ms_dispatch 0x7f8cb8000b90 osd_map(1..1 src has 1..1) v4 2024-03-29T11:19:24.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.414+0000 7f8c9f7fe640 3 client.4113.objecter handle_osd_map got epochs [1,1] > 0 2024-03-29T11:19:24.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.414+0000 7f8c9f7fe640 3 client.4113.objecter handle_osd_map decoding full epoch 1 2024-03-29T11:19:24.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.414+0000 7f8c9f7fe640 20 client.4113.objecter dump_active .. 0 homeless 2024-03-29T11:19:24.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.415+0000 7f8cbd11c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:24.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.415+0000 7f8cbd11c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:24.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.415+0000 7f8cbd11c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:19:24.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.415+0000 7f8cbd11c640 1 -- 172.21.15.104:0/3661510795 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c98005080 con 0x7f8ca400aab0 2024-03-29T11:19:24.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.418+0000 7f8c9f7fe640 1 -- 172.21.15.104:0/3661510795 <== mon.0 v1:172.21.15.104:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f8cac005780 con 0x7f8ca400aab0 2024-03-29T11:19:24.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.418+0000 7f8c9f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:24.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.418+0000 7f8c9f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:24.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.486+0000 7f8cbd11c640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:19:24.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.486+0000 7f8cbd11c640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:19:24.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.486+0000 7f8cbd11c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:19:24.463 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.486+0000 7f8cbd11c640 1 -- 172.21.15.104:0/3661510795 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f8c98005330 con 0x7f8ca400aab0 2024-03-29T11:19:24.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.488+0000 7f8c9f7fe640 1 -- 172.21.15.104:0/3661510795 <== mon.0 v1:172.21.15.104:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f8cac0059a0 con 0x7f8ca400aab0 2024-03-29T11:19:24.464 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.488+0000 7f8c9f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:19:24.465 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.488+0000 7f8c9f7fe640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-29T11:19:24.465 INFO:teuthology.orchestra.run.smithi104.stderr:adjusted tunables profile to default 2024-03-29T11:19:24.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.489+0000 7f8cbd11c640 10 librados: watch_flush enter 2024-03-29T11:19:24.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.489+0000 7f8cbd11c640 10 librados: watch_flush exit 2024-03-29T11:19:24.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.489+0000 7f8cbd11c640 20 client.4113.objecter shutdown clearing up homeless session... 2024-03-29T11:19:24.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.489+0000 7f8cbd11c640 10 client.4113.objecter successfully canceled tick 2024-03-29T11:19:24.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.489+0000 7f8cbd11c640 10 monclient: shutdown 2024-03-29T11:19:24.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.489+0000 7f8cbd11c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:24.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.489+0000 7f8cbd11c640 1 -- 172.21.15.104:0/3661510795 >> v1:172.21.15.104:6789/0 conn(0x7f8ca400aab0 legacy=0x7f8cb8138fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:24.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.489+0000 7f8cbd11c640 1 -- 172.21.15.104:0/3661510795 shutdown_connections 2024-03-29T11:19:24.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.489+0000 7f8cbd11c640 1 -- 172.21.15.104:0/3661510795 >> 172.21.15.104:0/3661510795 conn(0x7f8cb8066b00 msgr2=0x7f8cb806b230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:24.466 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.489+0000 7f8cbd11c640 1 -- 172.21.15.104:0/3661510795 shutdown_connections 2024-03-29T11:19:24.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.489+0000 7f8cbd11c640 1 -- 172.21.15.104:0/3661510795 wait complete. 2024-03-29T11:19:24.467 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:24.489+0000 7f8cbd11c640 1 librados: shutdown 2024-03-29T11:19:24.477 INFO:tasks.ceph:check_enable_crimson: False 2024-03-29T11:19:24.477 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T11:19:24.477 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-29T11:19:24.477 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T11:19:24.505 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-29T11:19:24.505 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T11:19:24.572 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-29T11:19:24.572 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T11:19:24.639 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-03-29T11:19:24.639 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-29T11:19:24.707 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-29T11:19:24.707 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-29T11:19:24.734 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-29T11:19:24.734 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-29T11:19:24.803 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-29T11:19:24.803 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-29T11:19:24.865 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-03-29T11:19:24.889+0000 7f32f3443040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:19:24.871 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-03-29T11:19:24.871 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-29T11:19:24.895 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-29T11:19:24.919+0000 7f0ede043040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:19:24.921 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-03-29T11:19:24.945+0000 7f32f3443040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:19:24.938 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster ceph osd new 9aa5839f-9605-4fab-b090-d4c7de9f53ec 0 2024-03-29T11:19:24.948 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-29T11:19:24.971+0000 7f0ede043040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:19:25.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.100+0000 7f8deb631640 10 monclient: get_monmap_and_config 2024-03-29T11:19:25.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.100+0000 7f8deb631640 10 monclient: build_initial_monmap 2024-03-29T11:19:25.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.100+0000 7f8deb631640 10 monclient: monmap: 2024-03-29T11:19:25.078 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 0 2024-03-29T11:19:25.078 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:25.078 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:19:25.102114+0000 2024-03-29T11:19:25.078 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:19:25.102114+0000 2024-03-29T11:19:25.078 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:25.078 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:25.078 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:19:25.078 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:19:25.079 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:19:25.079 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:25.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.101+0000 7f8deb631640 1 Processor -- start 2024-03-29T11:19:25.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.101+0000 7f8deb631640 1 -- start start 2024-03-29T11:19:25.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.101+0000 7f8deb631640 10 monclient: init 2024-03-29T11:19:25.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8deb631640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:25.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8deb631640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:25.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8deb631640 10 monclient(hunting): picked mon.noname-a con 0x7f8de409ac40 addr v1:172.21.15.104:6789/0 2024-03-29T11:19:25.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8deb631640 10 monclient(hunting): picked mon.noname-c con 0x7f8de409c870 addr v1:172.21.15.119:6789/0 2024-03-29T11:19:25.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8deb631640 10 monclient(hunting): picked mon.noname-b con 0x7f8de40b4360 addr v1:172.21.15.104:6790/0 2024-03-29T11:19:25.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8deb631640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8de409dec0 con 0x7f8de409ac40 2024-03-29T11:19:25.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8deb631640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8de409da30 con 0x7f8de409c870 2024-03-29T11:19:25.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8deb631640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8de40b39c0 con 0x7f8de40b4360 2024-03-29T11:19:25.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8deb631640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:25.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8deb631640 10 monclient(hunting): authenticate will time out at 942.379333s 2024-03-29T11:19:25.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8de8bcc640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f8de409c870 0x7f8de413f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:36578/0 (socket says 172.21.15.119:36578) 2024-03-29T11:19:25.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8de8bcc640 1 -- 172.21.15.119:0/180649447 learned_addr learned my addr 172.21.15.119:0/180649447 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-03-29T11:19:25.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8ddbfff640 1 -- 172.21.15.119:0/180649447 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2856612877 0 0) 0x7f8de409da30 con 0x7f8de409c870 2024-03-29T11:19:25.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8ddbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:25.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8ddbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:25.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8ddbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:25.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8ddbfff640 10 monclient(hunting): my global_id is 4102 2024-03-29T11:19:25.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8ddbfff640 1 -- 172.21.15.119:0/180649447 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8dd0003520 con 0x7f8de409c870 2024-03-29T11:19:25.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8ddbfff640 1 -- 172.21.15.119:0/180649447 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 180922251 0 0) 0x7f8dd4001110 con 0x7f8de409ac40 2024-03-29T11:19:25.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8ddbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:25.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8ddbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:25.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8ddbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:25.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8ddbfff640 10 monclient(hunting): my global_id is 4116 2024-03-29T11:19:25.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8ddbfff640 1 -- 172.21.15.119:0/180649447 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8de409da30 con 0x7f8de409ac40 2024-03-29T11:19:25.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8ddbfff640 1 -- 172.21.15.119:0/180649447 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3296459931 0 0) 0x7f8de40b39c0 con 0x7f8de40b4360 2024-03-29T11:19:25.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8ddbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:25.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8ddbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:25.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8ddbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:25.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8ddbfff640 10 monclient(hunting): my global_id is 4112 2024-03-29T11:19:25.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8ddbfff640 1 -- 172.21.15.119:0/180649447 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8dd4001110 con 0x7f8de40b4360 2024-03-29T11:19:25.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8ddbfff640 1 -- 172.21.15.119:0/180649447 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2505010991 0 0) 0x7f8dd0003520 con 0x7f8de409c870 2024-03-29T11:19:25.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8ddbfff640 1 -- 172.21.15.119:0/180649447 >> v1:172.21.15.104:6790/0 conn(0x7f8de40b4360 legacy=0x7f8de414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:25.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8ddbfff640 1 -- 172.21.15.119:0/180649447 >> v1:172.21.15.104:6789/0 conn(0x7f8de409ac40 legacy=0x7f8de40b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:25.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8ddbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:19:25.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8ddbfff640 1 monclient: found mon.noname-c 2024-03-29T11:19:25.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8ddbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:25.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8ddbfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.119:6789/0 2024-03-29T11:19:25.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8ddbfff640 1 -- 172.21.15.119:0/180649447 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de40b5160 con 0x7f8de409c870 2024-03-29T11:19:25.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8ddbfff640 10 monclient: _finish_auth 0 2024-03-29T11:19:25.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8ddbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:19:25.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8ddbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:25.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8deb631640 5 monclient: authenticate success, global_id 4102 2024-03-29T11:19:25.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8ddbfff640 1 -- 172.21.15.119:0/180649447 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f8de0001690 con 0x7f8de409c870 2024-03-29T11:19:25.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8ddbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:25.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8ddbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:25.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8ddbfff640 10 monclient: dump: 2024-03-29T11:19:25.091 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 1 2024-03-29T11:19:25.091 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:25.091 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.091 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.091 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:25.091 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:25.091 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:25.091 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:25.091 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:25.091 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:25.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8deb631640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:25.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8ddbfff640 1 -- 172.21.15.119:0/180649447 <== mon.1 v1:172.21.15.119:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8de0001870 con 0x7f8de409c870 2024-03-29T11:19:25.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8ddbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:25.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8deb631640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:25.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8ddbfff640 1 -- 172.21.15.119:0/180649447 <== mon.1 v1:172.21.15.119:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f8de0001c80 con 0x7f8de409c870 2024-03-29T11:19:25.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8ddbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:25.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8deb631640 4 set_mon_vals no callback set 2024-03-29T11:19:25.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8ddbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:25.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.102+0000 7f8ddbfff640 10 monclient: dump: 2024-03-29T11:19:25.092 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 1 2024-03-29T11:19:25.092 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:25.092 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.092 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.092 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:25.092 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:25.092 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:25.092 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:25.092 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:25.092 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:25.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.103+0000 7f8deb631640 10 monclient: shutdown 2024-03-29T11:19:25.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.103+0000 7f8deb631640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:25.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.103+0000 7f8deb631640 1 -- 172.21.15.119:0/180649447 >> v1:172.21.15.119:6789/0 conn(0x7f8de409c870 legacy=0x7f8de413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:25.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.103+0000 7f8deb631640 1 -- 172.21.15.119:0/180649447 shutdown_connections 2024-03-29T11:19:25.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.103+0000 7f8deb631640 1 -- 172.21.15.119:0/180649447 >> 172.21.15.119:0/180649447 conn(0x7f8de4066b00 msgr2=0x7f8de4066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:25.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.103+0000 7f8deb631640 1 -- 172.21.15.119:0/180649447 shutdown_connections 2024-03-29T11:19:25.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.103+0000 7f8deb631640 1 -- 172.21.15.119:0/180649447 wait complete. 2024-03-29T11:19:25.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.104+0000 7f8deb631640 10 monclient: build_initial_monmap 2024-03-29T11:19:25.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.104+0000 7f8deb631640 10 monclient: monmap: 2024-03-29T11:19:25.093 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 0 2024-03-29T11:19:25.093 INFO:teuthology.orchestra.run.smithi119.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:25.093 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 0.000000 2024-03-29T11:19:25.093 INFO:teuthology.orchestra.run.smithi119.stderr:created 0.000000 2024-03-29T11:19:25.093 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:25.093 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:25.093 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:19:25.093 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:19:25.093 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:19:25.093 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:25.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.104+0000 7f8deb631640 1 librados: starting msgr at 2024-03-29T11:19:25.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.104+0000 7f8deb631640 1 librados: starting objecter 2024-03-29T11:19:25.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.104+0000 7f8deb631640 1 Processor -- start 2024-03-29T11:19:25.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.104+0000 7f8deb631640 1 -- start start 2024-03-29T11:19:25.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.104+0000 7f8deb631640 1 librados: setting wanted keys 2024-03-29T11:19:25.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.104+0000 7f8deb631640 1 librados: calling monclient init 2024-03-29T11:19:25.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.104+0000 7f8deb631640 10 monclient: init 2024-03-29T11:19:25.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8deb631640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:25.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8deb631640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:25.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8deb631640 10 monclient(hunting): picked mon.noname-c con 0x7f8de409ac40 addr v1:172.21.15.104:6790/0 2024-03-29T11:19:25.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8deb631640 10 monclient(hunting): picked mon.noname-b con 0x7f8de41d7bc0 addr v1:172.21.15.119:6789/0 2024-03-29T11:19:25.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8deb631640 10 monclient(hunting): picked mon.noname-a con 0x7f8de40b17f0 addr v1:172.21.15.104:6789/0 2024-03-29T11:19:25.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8deb631640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8de413b890 con 0x7f8de40b17f0 2024-03-29T11:19:25.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8deb631640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8de41d76a0 con 0x7f8de41d7bc0 2024-03-29T11:19:25.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8deb631640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8de41406b0 con 0x7f8de409ac40 2024-03-29T11:19:25.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8deb631640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:25.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8deb631640 10 monclient(hunting): authenticate will time out at 942.382165s 2024-03-29T11:19:25.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8de8bcc640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f8de41d7bc0 0x7f8de413aa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:36582/0 (socket says 172.21.15.119:36582) 2024-03-29T11:19:25.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8de8bcc640 1 -- 172.21.15.119:0/152809885 learned_addr learned my addr 172.21.15.119:0/152809885 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-03-29T11:19:25.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 10 client.?.objecter ms_handle_connect 0x7f8de41d7bc0 2024-03-29T11:19:25.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:25.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 10 client.?.objecter ms_handle_connect 0x7f8de40b17f0 2024-03-29T11:19:25.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:25.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 10 client.?.objecter ms_handle_connect 0x7f8de409ac40 2024-03-29T11:19:25.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:25.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 1 -- 172.21.15.119:0/152809885 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2414790526 0 0) 0x7f8de41d76a0 con 0x7f8de41d7bc0 2024-03-29T11:19:25.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:25.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:25.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:25.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 10 monclient(hunting): my global_id is 4105 2024-03-29T11:19:25.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 1 -- 172.21.15.119:0/152809885 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8dc8004eb0 con 0x7f8de41d7bc0 2024-03-29T11:19:25.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 1 -- 172.21.15.119:0/152809885 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 143889216 0 0) 0x7f8de413b890 con 0x7f8de40b17f0 2024-03-29T11:19:25.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:25.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:25.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:25.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 10 monclient(hunting): my global_id is 4119 2024-03-29T11:19:25.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 1 -- 172.21.15.119:0/152809885 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8de41d76a0 con 0x7f8de40b17f0 2024-03-29T11:19:25.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 1 -- 172.21.15.119:0/152809885 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3078697123 0 0) 0x7f8de41406b0 con 0x7f8de409ac40 2024-03-29T11:19:25.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:25.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:25.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:25.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 10 monclient(hunting): my global_id is 4115 2024-03-29T11:19:25.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 1 -- 172.21.15.119:0/152809885 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8de413b890 con 0x7f8de409ac40 2024-03-29T11:19:25.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 1 -- 172.21.15.119:0/152809885 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 413323296 0 0) 0x7f8dc8004eb0 con 0x7f8de41d7bc0 2024-03-29T11:19:25.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 1 -- 172.21.15.119:0/152809885 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8de41406b0 con 0x7f8de41d7bc0 2024-03-29T11:19:25.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 1 -- 172.21.15.119:0/152809885 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f8de0001bc0 con 0x7f8de41d7bc0 2024-03-29T11:19:25.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:25.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:25.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 10 monclient(hunting): dump: 2024-03-29T11:19:25.096 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 1 2024-03-29T11:19:25.097 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:25.097 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.097 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.097 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:25.097 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:25.097 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:25.097 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:25.097 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:25.097 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:25.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:25.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 1 -- 172.21.15.119:0/152809885 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 719697833 0 0) 0x7f8de41d76a0 con 0x7f8de40b17f0 2024-03-29T11:19:25.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 1 -- 172.21.15.119:0/152809885 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8dc8004eb0 con 0x7f8de40b17f0 2024-03-29T11:19:25.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 1 -- 172.21.15.119:0/152809885 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f8ddc001b50 con 0x7f8de40b17f0 2024-03-29T11:19:25.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:25.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:25.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 10 monclient(hunting): dump: 2024-03-29T11:19:25.097 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 1 2024-03-29T11:19:25.098 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:25.098 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.098 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.098 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:25.098 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:25.098 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:25.098 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:25.098 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:25.098 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:25.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 1 -- 172.21.15.119:0/152809885 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1527024834 0 0) 0x7f8de413b890 con 0x7f8de409ac40 2024-03-29T11:19:25.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 1 -- 172.21.15.119:0/152809885 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8de41d76a0 con 0x7f8de409ac40 2024-03-29T11:19:25.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 1 -- 172.21.15.119:0/152809885 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f8dd40018a0 con 0x7f8de409ac40 2024-03-29T11:19:25.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:25.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:25.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 10 monclient(hunting): dump: 2024-03-29T11:19:25.099 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 1 2024-03-29T11:19:25.099 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:25.099 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.099 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.099 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:25.099 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:25.099 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:25.099 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:25.099 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:25.099 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:25.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 1 -- 172.21.15.119:0/152809885 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4239120983 0 0) 0x7f8de41406b0 con 0x7f8de41d7bc0 2024-03-29T11:19:25.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 1 -- 172.21.15.119:0/152809885 >> v1:172.21.15.104:6790/0 conn(0x7f8de409ac40 legacy=0x7f8de413ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:25.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 1 -- 172.21.15.119:0/152809885 >> v1:172.21.15.104:6789/0 conn(0x7f8de40b17f0 legacy=0x7f8de413b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:25.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:19:25.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 1 monclient: found mon.b 2024-03-29T11:19:25.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:25.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:19:25.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 1 -- 172.21.15.119:0/152809885 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de4140840 con 0x7f8de41d7bc0 2024-03-29T11:19:25.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:19:25.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:19:25.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8dd9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:25.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8deb631640 5 monclient: authenticate success, global_id 4105 2024-03-29T11:19:25.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8deb631640 10 monclient: _renew_subs 2024-03-29T11:19:25.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8deb631640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:19:25.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.105+0000 7f8deb631640 1 -- 172.21.15.119:0/152809885 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8de40b6f40 con 0x7f8de41d7bc0 2024-03-29T11:19:25.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.106+0000 7f8dd9ffb640 1 -- 172.21.15.119:0/152809885 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8de0004bf0 con 0x7f8de41d7bc0 2024-03-29T11:19:25.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.106+0000 7f8dd9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:25.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.106+0000 7f8dd9ffb640 1 -- 172.21.15.119:0/152809885 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f8de0004f00 con 0x7f8de41d7bc0 2024-03-29T11:19:25.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.106+0000 7f8dd9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:25.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.106+0000 7f8dd9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:25.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.106+0000 7f8dd9ffb640 10 monclient: dump: 2024-03-29T11:19:25.100 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 1 2024-03-29T11:19:25.100 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:25.101 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.101 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.101 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:25.101 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:25.101 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:25.101 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:25.101 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:25.101 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:25.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.106+0000 7f8ddb7fe640 4 set_mon_vals no callback set 2024-03-29T11:19:25.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.106+0000 7f8deb631640 10 client.4105.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:25.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.106+0000 7f8deb631640 10 monclient: _renew_subs 2024-03-29T11:19:25.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.106+0000 7f8dd9ffb640 1 -- 172.21.15.119:0/152809885 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f8de0005470 con 0x7f8de41d7bc0 2024-03-29T11:19:25.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.106+0000 7f8deb631640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:19:25.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.106+0000 7f8deb631640 1 -- 172.21.15.119:0/152809885 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8de41d6f40 con 0x7f8de41d7bc0 2024-03-29T11:19:25.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.106+0000 7f8deb631640 1 librados: init done 2024-03-29T11:19:25.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.106+0000 7f8dd9ffb640 1 -- 172.21.15.119:0/152809885 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 2533028071 0 0) 0x7f8de00011f0 con 0x7f8de41d7bc0 2024-03-29T11:19:25.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.106+0000 7f8dd9ffb640 10 client.4105.objecter ms_dispatch 0x7f8de4000b90 osd_map(2..2 src has 1..2) v4 2024-03-29T11:19:25.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.106+0000 7f8dd9ffb640 3 client.4105.objecter handle_osd_map got epochs [2,2] > 0 2024-03-29T11:19:25.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.106+0000 7f8dd9ffb640 3 client.4105.objecter handle_osd_map decoding full epoch 2 2024-03-29T11:19:25.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.106+0000 7f8dd9ffb640 20 client.4105.objecter dump_active .. 0 homeless 2024-03-29T11:19:25.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.106+0000 7f8deb631640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:25.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.106+0000 7f8deb631640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:25.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.106+0000 7f8deb631640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:19:25.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.106+0000 7f8deb631640 1 -- 172.21.15.119:0/152809885 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8de409b020 con 0x7f8de41d7bc0 2024-03-29T11:19:25.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.109+0000 7f8dd9ffb640 1 -- 172.21.15.119:0/152809885 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f8de0005b10 con 0x7f8de41d7bc0 2024-03-29T11:19:25.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.109+0000 7f8dd9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:25.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.109+0000 7f8dd9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:25.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.180+0000 7f8deb631640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "9aa5839f-9605-4fab-b090-d4c7de9f53ec", "id": 0}] 2024-03-29T11:19:25.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.180+0000 7f8deb631640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "9aa5839f-9605-4fab-b090-d4c7de9f53ec", "id": 0}] 2024-03-29T11:19:25.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.180+0000 7f8deb631640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:19:25.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.180+0000 7f8deb631640 1 -- 172.21.15.119:0/152809885 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "osd new", "uuid": "9aa5839f-9605-4fab-b090-d4c7de9f53ec", "id": 0} v 0) v1 -- 0x7f8de404f730 con 0x7f8de41d7bc0 2024-03-29T11:19:25.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.186+0000 7f8dd9ffb640 1 -- 172.21.15.119:0/152809885 <== mon.1 v1:172.21.15.119:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9aa5839f-9605-4fab-b090-d4c7de9f53ec", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 3787462616 0 732707515) 0x7f8de0005da0 con 0x7f8de41d7bc0 2024-03-29T11:19:25.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.186+0000 7f8dd9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "9aa5839f-9605-4fab-b090-d4c7de9f53ec", "id": 0}] 2024-03-29T11:19:25.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.186+0000 7f8dd9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:25.163 INFO:teuthology.orchestra.run.smithi119.stdout:0 2024-03-29T11:19:25.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.186+0000 7f8deb631640 10 librados: watch_flush enter 2024-03-29T11:19:25.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.186+0000 7f8deb631640 10 librados: watch_flush exit 2024-03-29T11:19:25.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.186+0000 7f8deb631640 20 client.4105.objecter shutdown clearing up homeless session... 2024-03-29T11:19:25.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.186+0000 7f8deb631640 10 client.4105.objecter successfully canceled tick 2024-03-29T11:19:25.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.186+0000 7f8deb631640 10 monclient: shutdown 2024-03-29T11:19:25.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.186+0000 7f8deb631640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:25.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.186+0000 7f8deb631640 1 -- 172.21.15.119:0/152809885 >> v1:172.21.15.119:6789/0 conn(0x7f8de41d7bc0 legacy=0x7f8de413aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:25.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.186+0000 7f8deb631640 1 -- 172.21.15.119:0/152809885 shutdown_connections 2024-03-29T11:19:25.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.186+0000 7f8deb631640 1 -- 172.21.15.119:0/152809885 >> 172.21.15.119:0/152809885 conn(0x7f8de4066b00 msgr2=0x7f8de413d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:25.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.187+0000 7f8deb631640 1 -- 172.21.15.119:0/152809885 shutdown_connections 2024-03-29T11:19:25.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.187+0000 7f8deb631640 1 -- 172.21.15.119:0/152809885 wait complete. 2024-03-29T11:19:25.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.187+0000 7f8deb631640 1 librados: shutdown 2024-03-29T11:19:25.174 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster ceph osd new ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c 1 2024-03-29T11:19:25.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.295+0000 7fc27bb6b640 10 monclient: get_monmap_and_config 2024-03-29T11:19:25.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.295+0000 7fc27bb6b640 10 monclient: build_initial_monmap 2024-03-29T11:19:25.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.295+0000 7fc27bb6b640 10 monclient: monmap: 2024-03-29T11:19:25.273 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 0 2024-03-29T11:19:25.273 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:25.273 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:19:25.297084+0000 2024-03-29T11:19:25.273 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:19:25.297084+0000 2024-03-29T11:19:25.273 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:25.273 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:25.273 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:19:25.273 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:19:25.273 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:19:25.273 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:25.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.296+0000 7fc27bb6b640 1 Processor -- start 2024-03-29T11:19:25.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.296+0000 7fc27bb6b640 1 -- start start 2024-03-29T11:19:25.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.296+0000 7fc27bb6b640 10 monclient: init 2024-03-29T11:19:25.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc27bb6b640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:25.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc27bb6b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:19:25.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc27bb6b640 10 monclient(hunting): picked mon.noname-b con 0x7fc27409ac40 addr v1:172.21.15.104:6790/0 2024-03-29T11:19:25.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc27bb6b640 10 monclient(hunting): picked mon.noname-a con 0x7fc27409c870 addr v1:172.21.15.104:6789/0 2024-03-29T11:19:25.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc27bb6b640 10 monclient(hunting): picked mon.noname-c con 0x7fc2740b4360 addr v1:172.21.15.119:6789/0 2024-03-29T11:19:25.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc27bb6b640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc27409dec0 con 0x7fc27409c870 2024-03-29T11:19:25.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc27bb6b640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc27409da30 con 0x7fc2740b4360 2024-03-29T11:19:25.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc27bb6b640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2740b39c0 con 0x7fc27409ac40 2024-03-29T11:19:25.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc27bb6b640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:25.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc27bb6b640 10 monclient(hunting): authenticate will time out at 942.574265s 2024-03-29T11:19:25.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc27a108640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fc2740b4360 0x7fc27414d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:36584/0 (socket says 172.21.15.119:36584) 2024-03-29T11:19:25.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc27a108640 1 -- 172.21.15.119:0/3198169628 learned_addr learned my addr 172.21.15.119:0/3198169628 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-03-29T11:19:25.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc278905640 1 -- 172.21.15.119:0/3198169628 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 215603144 0 0) 0x7fc27409da30 con 0x7fc2740b4360 2024-03-29T11:19:25.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc278905640 10 monclient(hunting): _negotiate 2024-03-29T11:19:25.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc278905640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:25.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc278905640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:25.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc278905640 10 monclient(hunting): my global_id is 4108 2024-03-29T11:19:25.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc278905640 1 -- 172.21.15.119:0/3198169628 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc260003520 con 0x7fc2740b4360 2024-03-29T11:19:25.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc278905640 1 -- 172.21.15.119:0/3198169628 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2387132886 0 0) 0x7fc270001040 con 0x7fc27409c870 2024-03-29T11:19:25.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc278905640 10 monclient(hunting): _negotiate 2024-03-29T11:19:25.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc278905640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:25.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc278905640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:25.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc278905640 10 monclient(hunting): my global_id is 4122 2024-03-29T11:19:25.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc278905640 1 -- 172.21.15.119:0/3198169628 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc27409da30 con 0x7fc27409c870 2024-03-29T11:19:25.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc278905640 1 -- 172.21.15.119:0/3198169628 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2823363432 0 0) 0x7fc2740b39c0 con 0x7fc27409ac40 2024-03-29T11:19:25.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc278905640 10 monclient(hunting): _negotiate 2024-03-29T11:19:25.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc278905640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:25.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc278905640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:25.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc278905640 10 monclient(hunting): my global_id is 4118 2024-03-29T11:19:25.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc278905640 1 -- 172.21.15.119:0/3198169628 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc270001040 con 0x7fc27409ac40 2024-03-29T11:19:25.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc278905640 1 -- 172.21.15.119:0/3198169628 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3446001633 0 0) 0x7fc260003520 con 0x7fc2740b4360 2024-03-29T11:19:25.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc278905640 1 -- 172.21.15.119:0/3198169628 >> v1:172.21.15.104:6790/0 conn(0x7fc27409ac40 legacy=0x7fc2740b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:25.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc278905640 1 -- 172.21.15.119:0/3198169628 >> v1:172.21.15.104:6789/0 conn(0x7fc27409c870 legacy=0x7fc27413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:25.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc278905640 10 monclient: _finish_hunting 0 2024-03-29T11:19:25.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc278905640 1 monclient: found mon.noname-c 2024-03-29T11:19:25.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc278905640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:25.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc278905640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.119:6789/0 2024-03-29T11:19:25.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc278905640 1 -- 172.21.15.119:0/3198169628 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2740b5160 con 0x7fc2740b4360 2024-03-29T11:19:25.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc278905640 10 monclient: _finish_auth 0 2024-03-29T11:19:25.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc278905640 10 monclient: _check_auth_tickets 2024-03-29T11:19:25.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc278905640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:25.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc278905640 1 -- 172.21.15.119:0/3198169628 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fc26c001a10 con 0x7fc2740b4360 2024-03-29T11:19:25.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc278905640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:25.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc278905640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:25.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc278905640 10 monclient: dump: 2024-03-29T11:19:25.285 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 1 2024-03-29T11:19:25.285 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:25.285 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.285 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.285 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:25.285 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:25.286 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:25.286 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:25.286 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:25.286 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:25.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc27bb6b640 5 monclient: authenticate success, global_id 4108 2024-03-29T11:19:25.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc27bb6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:25.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc278905640 1 -- 172.21.15.119:0/3198169628 <== mon.1 v1:172.21.15.119:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc26c001b70 con 0x7fc2740b4360 2024-03-29T11:19:25.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc278905640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:25.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc278905640 1 -- 172.21.15.119:0/3198169628 <== mon.1 v1:172.21.15.119:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fc26c001f80 con 0x7fc2740b4360 2024-03-29T11:19:25.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc27bb6b640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:25.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc27bb6b640 4 set_mon_vals no callback set 2024-03-29T11:19:25.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.297+0000 7fc278905640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:25.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.298+0000 7fc278905640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:25.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.298+0000 7fc278905640 10 monclient: dump: 2024-03-29T11:19:25.287 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 1 2024-03-29T11:19:25.287 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:25.287 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.287 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.287 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:25.287 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:25.287 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:25.287 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:25.287 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:25.287 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:25.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.298+0000 7fc27bb6b640 10 monclient: shutdown 2024-03-29T11:19:25.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.298+0000 7fc27bb6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:25.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.298+0000 7fc27bb6b640 1 -- 172.21.15.119:0/3198169628 >> v1:172.21.15.119:6789/0 conn(0x7fc2740b4360 legacy=0x7fc27414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:25.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.298+0000 7fc27bb6b640 1 -- 172.21.15.119:0/3198169628 shutdown_connections 2024-03-29T11:19:25.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.298+0000 7fc27bb6b640 1 -- 172.21.15.119:0/3198169628 >> 172.21.15.119:0/3198169628 conn(0x7fc274066b00 msgr2=0x7fc274066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:25.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.298+0000 7fc27bb6b640 1 -- 172.21.15.119:0/3198169628 shutdown_connections 2024-03-29T11:19:25.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.298+0000 7fc27bb6b640 1 -- 172.21.15.119:0/3198169628 wait complete. 2024-03-29T11:19:25.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.299+0000 7fc27bb6b640 10 monclient: build_initial_monmap 2024-03-29T11:19:25.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.299+0000 7fc27bb6b640 10 monclient: monmap: 2024-03-29T11:19:25.288 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 0 2024-03-29T11:19:25.288 INFO:teuthology.orchestra.run.smithi119.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:25.288 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 0.000000 2024-03-29T11:19:25.288 INFO:teuthology.orchestra.run.smithi119.stderr:created 0.000000 2024-03-29T11:19:25.288 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:25.288 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:25.288 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:19:25.288 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:19:25.288 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:19:25.288 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:25.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.299+0000 7fc27bb6b640 1 librados: starting msgr at 2024-03-29T11:19:25.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.299+0000 7fc27bb6b640 1 librados: starting objecter 2024-03-29T11:19:25.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.299+0000 7fc27bb6b640 1 Processor -- start 2024-03-29T11:19:25.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.299+0000 7fc27bb6b640 1 -- start start 2024-03-29T11:19:25.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.299+0000 7fc27bb6b640 1 librados: setting wanted keys 2024-03-29T11:19:25.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.299+0000 7fc27bb6b640 1 librados: calling monclient init 2024-03-29T11:19:25.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.299+0000 7fc27bb6b640 10 monclient: init 2024-03-29T11:19:25.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc27bb6b640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:25.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc27bb6b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:25.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc27bb6b640 10 monclient(hunting): picked mon.noname-a con 0x7fc27409ac40 addr v1:172.21.15.104:6789/0 2024-03-29T11:19:25.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc27bb6b640 10 monclient(hunting): picked mon.noname-b con 0x7fc2741d7cd0 addr v1:172.21.15.119:6789/0 2024-03-29T11:19:25.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc27bb6b640 10 monclient(hunting): picked mon.noname-c con 0x7fc2740ad1e0 addr v1:172.21.15.104:6790/0 2024-03-29T11:19:25.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc27bb6b640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2741d7130 con 0x7fc27409ac40 2024-03-29T11:19:25.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc27bb6b640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc274139b10 con 0x7fc2741d7cd0 2024-03-29T11:19:25.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc27bb6b640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc27413ffa0 con 0x7fc2740ad1e0 2024-03-29T11:19:25.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc27bb6b640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:25.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc27bb6b640 10 monclient(hunting): authenticate will time out at 942.577218s 2024-03-29T11:19:25.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc279106640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fc2741d7cd0 0x7fc274138cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:36586/0 (socket says 172.21.15.119:36586) 2024-03-29T11:19:25.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc279106640 1 -- 172.21.15.119:0/1622841074 learned_addr learned my addr 172.21.15.119:0/1622841074 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-03-29T11:19:25.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 10 client.?.objecter ms_handle_connect 0x7fc2741d7cd0 2024-03-29T11:19:25.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:25.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 10 client.?.objecter ms_handle_connect 0x7fc27409ac40 2024-03-29T11:19:25.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:25.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 10 client.?.objecter ms_handle_connect 0x7fc2740ad1e0 2024-03-29T11:19:25.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:25.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 1 -- 172.21.15.119:0/1622841074 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4143294449 0 0) 0x7fc274139b10 con 0x7fc2741d7cd0 2024-03-29T11:19:25.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:25.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:25.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:25.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 10 monclient(hunting): my global_id is 4111 2024-03-29T11:19:25.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 1 -- 172.21.15.119:0/1622841074 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc258004eb0 con 0x7fc2741d7cd0 2024-03-29T11:19:25.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 1 -- 172.21.15.119:0/1622841074 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3135886562 0 0) 0x7fc2741d7130 con 0x7fc27409ac40 2024-03-29T11:19:25.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:25.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:25.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:25.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 10 monclient(hunting): my global_id is 4125 2024-03-29T11:19:25.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 1 -- 172.21.15.119:0/1622841074 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc274139b10 con 0x7fc27409ac40 2024-03-29T11:19:25.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 1 -- 172.21.15.119:0/1622841074 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1711734697 0 0) 0x7fc27413ffa0 con 0x7fc2740ad1e0 2024-03-29T11:19:25.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:25.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:25.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:25.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 10 monclient(hunting): my global_id is 4121 2024-03-29T11:19:25.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 1 -- 172.21.15.119:0/1622841074 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2741d7130 con 0x7fc2740ad1e0 2024-03-29T11:19:25.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 1 -- 172.21.15.119:0/1622841074 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2647623492 0 0) 0x7fc274139b10 con 0x7fc27409ac40 2024-03-29T11:19:25.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 1 -- 172.21.15.119:0/1622841074 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc27413ffa0 con 0x7fc27409ac40 2024-03-29T11:19:25.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 1 -- 172.21.15.119:0/1622841074 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fc264001870 con 0x7fc27409ac40 2024-03-29T11:19:25.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:25.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:25.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 10 monclient(hunting): dump: 2024-03-29T11:19:25.292 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 1 2024-03-29T11:19:25.292 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:25.292 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.292 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.292 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:25.292 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:25.292 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:25.292 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:25.292 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:25.292 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:25.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:25.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 1 -- 172.21.15.119:0/1622841074 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1931690565 0 0) 0x7fc258004eb0 con 0x7fc2741d7cd0 2024-03-29T11:19:25.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 1 -- 172.21.15.119:0/1622841074 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc274139b10 con 0x7fc2741d7cd0 2024-03-29T11:19:25.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 1 -- 172.21.15.119:0/1622841074 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fc270001950 con 0x7fc2741d7cd0 2024-03-29T11:19:25.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:25.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:25.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 10 monclient(hunting): dump: 2024-03-29T11:19:25.293 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 1 2024-03-29T11:19:25.293 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:25.293 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.293 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.293 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:25.293 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:25.293 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:25.293 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:25.293 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:25.293 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:25.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 1 -- 172.21.15.119:0/1622841074 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2170595092 0 0) 0x7fc2741d7130 con 0x7fc2740ad1e0 2024-03-29T11:19:25.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 1 -- 172.21.15.119:0/1622841074 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc258004eb0 con 0x7fc2740ad1e0 2024-03-29T11:19:25.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 1 -- 172.21.15.119:0/1622841074 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fc26c001e80 con 0x7fc2740ad1e0 2024-03-29T11:19:25.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:25.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:25.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 10 monclient(hunting): dump: 2024-03-29T11:19:25.294 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 1 2024-03-29T11:19:25.294 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:25.294 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.294 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.294 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:25.294 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:25.294 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:25.294 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:25.294 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:25.294 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:25.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 1 -- 172.21.15.119:0/1622841074 <== mon.0 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3457239779 0 0) 0x7fc27413ffa0 con 0x7fc27409ac40 2024-03-29T11:19:25.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 1 -- 172.21.15.119:0/1622841074 >> v1:172.21.15.104:6790/0 conn(0x7fc2740ad1e0 legacy=0x7fc274139400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:25.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 1 -- 172.21.15.119:0/1622841074 >> v1:172.21.15.119:6789/0 conn(0x7fc2741d7cd0 legacy=0x7fc274138cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:25.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:19:25.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 1 monclient: found mon.a 2024-03-29T11:19:25.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:25.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:19:25.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 1 -- 172.21.15.119:0/1622841074 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2740ae520 con 0x7fc27409ac40 2024-03-29T11:19:25.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:19:25.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:19:25.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc26a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:25.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc27bb6b640 5 monclient: authenticate success, global_id 4125 2024-03-29T11:19:25.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc27bb6b640 10 monclient: _renew_subs 2024-03-29T11:19:25.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc27bb6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:19:25.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.300+0000 7fc27bb6b640 1 -- 172.21.15.119:0/1622841074 --> v1:172.21.15.104:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2740ae800 con 0x7fc27409ac40 2024-03-29T11:19:25.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.301+0000 7fc27bb6b640 10 client.4125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:25.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.301+0000 7fc27bb6b640 10 monclient: _renew_subs 2024-03-29T11:19:25.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.301+0000 7fc27bb6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:19:25.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.301+0000 7fc27bb6b640 1 -- 172.21.15.119:0/1622841074 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc274142b70 con 0x7fc27409ac40 2024-03-29T11:19:25.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.301+0000 7fc27bb6b640 1 librados: init done 2024-03-29T11:19:25.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.301+0000 7fc26a7fc640 1 -- 172.21.15.119:0/1622841074 <== mon.0 v1:172.21.15.104:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc264003610 con 0x7fc27409ac40 2024-03-29T11:19:25.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.301+0000 7fc26a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:25.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.301+0000 7fc26a7fc640 1 -- 172.21.15.119:0/1622841074 <== mon.0 v1:172.21.15.104:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fc264001d80 con 0x7fc27409ac40 2024-03-29T11:19:25.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.301+0000 7fc26a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:25.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.301+0000 7fc26a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:25.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.301+0000 7fc26bfff640 4 set_mon_vals no callback set 2024-03-29T11:19:25.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.301+0000 7fc26a7fc640 10 monclient: dump: 2024-03-29T11:19:25.296 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 1 2024-03-29T11:19:25.296 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:25.296 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.296 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.296 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:25.296 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:25.296 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:25.296 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:25.297 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:25.297 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:25.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.301+0000 7fc26a7fc640 1 -- 172.21.15.119:0/1622841074 <== mon.0 v1:172.21.15.104:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fc264003d30 con 0x7fc27409ac40 2024-03-29T11:19:25.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.301+0000 7fc26a7fc640 1 -- 172.21.15.119:0/1622841074 <== mon.0 v1:172.21.15.104:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 3967015899 0 0) 0x7fc2640053c0 con 0x7fc27409ac40 2024-03-29T11:19:25.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.301+0000 7fc26a7fc640 10 client.4125.objecter ms_dispatch 0x7fc274000b90 osd_map(3..3 src has 1..3) v4 2024-03-29T11:19:25.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.301+0000 7fc26a7fc640 3 client.4125.objecter handle_osd_map got epochs [3,3] > 0 2024-03-29T11:19:25.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.301+0000 7fc26a7fc640 3 client.4125.objecter handle_osd_map decoding full epoch 3 2024-03-29T11:19:25.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.301+0000 7fc26a7fc640 20 client.4125.objecter dump_active .. 0 homeless 2024-03-29T11:19:25.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.301+0000 7fc27bb6b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:25.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.301+0000 7fc27bb6b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:25.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.301+0000 7fc27bb6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:19:25.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.301+0000 7fc27bb6b640 1 -- 172.21.15.119:0/1622841074 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc25c005080 con 0x7fc27409ac40 2024-03-29T11:19:25.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.303+0000 7fc26a7fc640 1 -- 172.21.15.119:0/1622841074 <== mon.0 v1:172.21.15.104:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fc264005620 con 0x7fc27409ac40 2024-03-29T11:19:25.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.303+0000 7fc26a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:25.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.303+0000 7fc26a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:25.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.374+0000 7fc27bb6b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c", "id": 1}] 2024-03-29T11:19:25.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.374+0000 7fc27bb6b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c", "id": 1}] 2024-03-29T11:19:25.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.374+0000 7fc27bb6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:19:25.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.374+0000 7fc27bb6b640 1 -- 172.21.15.119:0/1622841074 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "osd new", "uuid": "ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c", "id": 1} v 0) v1 -- 0x7fc25c0058b0 con 0x7fc27409ac40 2024-03-29T11:19:25.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.376+0000 7fc26a7fc640 1 -- 172.21.15.119:0/1622841074 <== mon.0 v1:172.21.15.104:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 3748064303 0 940483276) 0x7fc2640057c0 con 0x7fc27409ac40 2024-03-29T11:19:25.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.376+0000 7fc26a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c", "id": 1}] 2024-03-29T11:19:25.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.376+0000 7fc26a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:25.353 INFO:teuthology.orchestra.run.smithi119.stdout:1 2024-03-29T11:19:25.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.376+0000 7fc27bb6b640 10 librados: watch_flush enter 2024-03-29T11:19:25.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.376+0000 7fc27bb6b640 10 librados: watch_flush exit 2024-03-29T11:19:25.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.376+0000 7fc27bb6b640 20 client.4125.objecter shutdown clearing up homeless session... 2024-03-29T11:19:25.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.376+0000 7fc27bb6b640 10 client.4125.objecter successfully canceled tick 2024-03-29T11:19:25.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.376+0000 7fc27bb6b640 10 monclient: shutdown 2024-03-29T11:19:25.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.376+0000 7fc27bb6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:25.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.377+0000 7fc27bb6b640 1 -- 172.21.15.119:0/1622841074 >> v1:172.21.15.104:6789/0 conn(0x7fc27409ac40 legacy=0x7fc2741d75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:25.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.377+0000 7fc27bb6b640 1 -- 172.21.15.119:0/1622841074 shutdown_connections 2024-03-29T11:19:25.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.377+0000 7fc27bb6b640 1 -- 172.21.15.119:0/1622841074 >> 172.21.15.119:0/1622841074 conn(0x7fc274066b00 msgr2=0x7fc2740b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:25.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.377+0000 7fc27bb6b640 1 -- 172.21.15.119:0/1622841074 shutdown_connections 2024-03-29T11:19:25.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.377+0000 7fc27bb6b640 1 -- 172.21.15.119:0/1622841074 wait complete. 2024-03-29T11:19:25.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.377+0000 7fc27bb6b640 1 librados: shutdown 2024-03-29T11:19:25.364 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster ceph osd new 9bcd508f-a564-47de-bc8d-d287c8a0441f 2 2024-03-29T11:19:25.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.487+0000 7f4cebfcd640 10 monclient: get_monmap_and_config 2024-03-29T11:19:25.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.487+0000 7f4cebfcd640 10 monclient: build_initial_monmap 2024-03-29T11:19:25.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.487+0000 7f4cebfcd640 10 monclient: monmap: 2024-03-29T11:19:25.465 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 0 2024-03-29T11:19:25.465 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:25.465 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:19:25.489156+0000 2024-03-29T11:19:25.465 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:19:25.489156+0000 2024-03-29T11:19:25.465 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:25.465 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:25.465 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:19:25.465 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:19:25.466 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:19:25.466 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:25.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.488+0000 7f4cebfcd640 1 Processor -- start 2024-03-29T11:19:25.473 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-29T11:19:25.495+0000 7f0ede043040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:19:25.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.488+0000 7f4cebfcd640 1 -- start start 2024-03-29T11:19:25.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.488+0000 7f4cebfcd640 10 monclient: init 2024-03-29T11:19:25.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.489+0000 7f4cebfcd640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:25.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.489+0000 7f4cebfcd640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:19:25.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.489+0000 7f4cebfcd640 10 monclient(hunting): picked mon.noname-b con 0x7f4ce409ac40 addr v1:172.21.15.104:6790/0 2024-03-29T11:19:25.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.489+0000 7f4cebfcd640 10 monclient(hunting): picked mon.noname-a con 0x7f4ce409c870 addr v1:172.21.15.104:6789/0 2024-03-29T11:19:25.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.489+0000 7f4cebfcd640 10 monclient(hunting): picked mon.noname-c con 0x7f4ce40b4360 addr v1:172.21.15.119:6789/0 2024-03-29T11:19:25.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.489+0000 7f4cebfcd640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ce409dec0 con 0x7f4ce409c870 2024-03-29T11:19:25.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.489+0000 7f4cebfcd640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ce409da30 con 0x7f4ce40b4360 2024-03-29T11:19:25.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.489+0000 7f4cebfcd640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ce40b39c0 con 0x7f4ce409ac40 2024-03-29T11:19:25.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.489+0000 7f4cebfcd640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:25.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.489+0000 7f4cebfcd640 10 monclient(hunting): authenticate will time out at 942.766348s 2024-03-29T11:19:25.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.489+0000 7f4cea56a640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f4ce40b4360 0x7f4ce414d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:36592/0 (socket says 172.21.15.119:36592) 2024-03-29T11:19:25.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.489+0000 7f4cea56a640 1 -- 172.21.15.119:0/3446613351 learned_addr learned my addr 172.21.15.119:0/3446613351 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-03-29T11:19:25.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.489+0000 7f4ce8d67640 1 -- 172.21.15.119:0/3446613351 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3318487700 0 0) 0x7f4ce409da30 con 0x7f4ce40b4360 2024-03-29T11:19:25.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.489+0000 7f4ce8d67640 10 monclient(hunting): _negotiate 2024-03-29T11:19:25.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.489+0000 7f4ce8d67640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:25.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.489+0000 7f4ce8d67640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:25.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.489+0000 7f4ce8d67640 10 monclient(hunting): my global_id is 4114 2024-03-29T11:19:25.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.489+0000 7f4ce8d67640 1 -- 172.21.15.119:0/3446613351 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cd0003520 con 0x7f4ce40b4360 2024-03-29T11:19:25.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.489+0000 7f4ce8d67640 1 -- 172.21.15.119:0/3446613351 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1804783304 0 0) 0x7f4cd4001110 con 0x7f4ce409c870 2024-03-29T11:19:25.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.489+0000 7f4ce8d67640 10 monclient(hunting): _negotiate 2024-03-29T11:19:25.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.489+0000 7f4ce8d67640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:25.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.489+0000 7f4ce8d67640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:25.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.489+0000 7f4ce8d67640 10 monclient(hunting): my global_id is 4128 2024-03-29T11:19:25.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.489+0000 7f4ce8d67640 1 -- 172.21.15.119:0/3446613351 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ce409da30 con 0x7f4ce409c870 2024-03-29T11:19:25.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.489+0000 7f4ce8d67640 1 -- 172.21.15.119:0/3446613351 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3350821524 0 0) 0x7f4ce40b39c0 con 0x7f4ce409ac40 2024-03-29T11:19:25.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.489+0000 7f4ce8d67640 10 monclient(hunting): _negotiate 2024-03-29T11:19:25.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.489+0000 7f4ce8d67640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:25.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.489+0000 7f4ce8d67640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:25.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.489+0000 7f4ce8d67640 10 monclient(hunting): my global_id is 4124 2024-03-29T11:19:25.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.489+0000 7f4ce8d67640 1 -- 172.21.15.119:0/3446613351 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cd4001110 con 0x7f4ce409ac40 2024-03-29T11:19:25.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.489+0000 7f4ce8d67640 1 -- 172.21.15.119:0/3446613351 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2908233831 0 0) 0x7f4cd0003520 con 0x7f4ce40b4360 2024-03-29T11:19:25.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.489+0000 7f4ce8d67640 1 -- 172.21.15.119:0/3446613351 >> v1:172.21.15.104:6790/0 conn(0x7f4ce409ac40 legacy=0x7f4ce40b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:25.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.489+0000 7f4ce8d67640 1 -- 172.21.15.119:0/3446613351 >> v1:172.21.15.104:6789/0 conn(0x7f4ce409c870 legacy=0x7f4ce413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:25.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.489+0000 7f4ce8d67640 10 monclient: _finish_hunting 0 2024-03-29T11:19:25.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.489+0000 7f4ce8d67640 1 monclient: found mon.noname-c 2024-03-29T11:19:25.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.489+0000 7f4ce8d67640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:25.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.489+0000 7f4ce8d67640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.119:6789/0 2024-03-29T11:19:25.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.489+0000 7f4ce8d67640 1 -- 172.21.15.119:0/3446613351 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce40b5160 con 0x7f4ce40b4360 2024-03-29T11:19:25.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.489+0000 7f4ce8d67640 10 monclient: _finish_auth 0 2024-03-29T11:19:25.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.489+0000 7f4ce8d67640 10 monclient: _check_auth_tickets 2024-03-29T11:19:25.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.489+0000 7f4ce8d67640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:25.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.489+0000 7f4ce8d67640 1 -- 172.21.15.119:0/3446613351 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f4cdc001a10 con 0x7f4ce40b4360 2024-03-29T11:19:25.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.489+0000 7f4cebfcd640 5 monclient: authenticate success, global_id 4114 2024-03-29T11:19:25.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.489+0000 7f4cebfcd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:25.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.489+0000 7f4ce8d67640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:25.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.489+0000 7f4ce8d67640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:25.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.489+0000 7f4ce8d67640 10 monclient: dump: 2024-03-29T11:19:25.478 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 1 2024-03-29T11:19:25.478 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:25.478 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.478 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.478 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:25.478 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:25.478 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:25.479 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:25.479 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:25.479 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:25.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.489+0000 7f4cebfcd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:25.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.490+0000 7f4ce8d67640 1 -- 172.21.15.119:0/3446613351 <== mon.1 v1:172.21.15.119:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4cdc001bf0 con 0x7f4ce40b4360 2024-03-29T11:19:25.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.490+0000 7f4ce8d67640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:25.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.490+0000 7f4ce8d67640 1 -- 172.21.15.119:0/3446613351 <== mon.1 v1:172.21.15.119:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f4cdc002000 con 0x7f4ce40b4360 2024-03-29T11:19:25.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.490+0000 7f4cebfcd640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:25.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.490+0000 7f4cebfcd640 4 set_mon_vals no callback set 2024-03-29T11:19:25.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.490+0000 7f4ce8d67640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:25.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.490+0000 7f4ce8d67640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:25.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.490+0000 7f4ce8d67640 10 monclient: dump: 2024-03-29T11:19:25.479 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 1 2024-03-29T11:19:25.479 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:25.479 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.479 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.479 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:25.480 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:25.480 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:25.480 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:25.480 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:25.480 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:25.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.490+0000 7f4cebfcd640 10 monclient: shutdown 2024-03-29T11:19:25.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.490+0000 7f4cebfcd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:25.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.490+0000 7f4cebfcd640 1 -- 172.21.15.119:0/3446613351 >> v1:172.21.15.119:6789/0 conn(0x7f4ce40b4360 legacy=0x7f4ce414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:25.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.490+0000 7f4cebfcd640 1 -- 172.21.15.119:0/3446613351 shutdown_connections 2024-03-29T11:19:25.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.490+0000 7f4cebfcd640 1 -- 172.21.15.119:0/3446613351 >> 172.21.15.119:0/3446613351 conn(0x7f4ce4066b00 msgr2=0x7f4ce4066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:25.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.490+0000 7f4cebfcd640 1 -- 172.21.15.119:0/3446613351 shutdown_connections 2024-03-29T11:19:25.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.490+0000 7f4cebfcd640 1 -- 172.21.15.119:0/3446613351 wait complete. 2024-03-29T11:19:25.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.491+0000 7f4cebfcd640 10 monclient: build_initial_monmap 2024-03-29T11:19:25.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.491+0000 7f4cebfcd640 10 monclient: monmap: 2024-03-29T11:19:25.480 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 0 2024-03-29T11:19:25.480 INFO:teuthology.orchestra.run.smithi119.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:25.481 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 0.000000 2024-03-29T11:19:25.481 INFO:teuthology.orchestra.run.smithi119.stderr:created 0.000000 2024-03-29T11:19:25.481 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:25.481 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:25.481 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:19:25.481 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:19:25.481 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:19:25.481 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:25.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.491+0000 7f4cebfcd640 1 librados: starting msgr at 2024-03-29T11:19:25.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.491+0000 7f4cebfcd640 1 librados: starting objecter 2024-03-29T11:19:25.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.491+0000 7f4cebfcd640 1 Processor -- start 2024-03-29T11:19:25.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.491+0000 7f4cebfcd640 1 -- start start 2024-03-29T11:19:25.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.491+0000 7f4cebfcd640 1 librados: setting wanted keys 2024-03-29T11:19:25.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.491+0000 7f4cebfcd640 1 librados: calling monclient init 2024-03-29T11:19:25.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.491+0000 7f4cebfcd640 10 monclient: init 2024-03-29T11:19:25.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cebfcd640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:25.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cebfcd640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:25.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cebfcd640 10 monclient(hunting): picked mon.noname-a con 0x7f4ce409ac40 addr v1:172.21.15.104:6789/0 2024-03-29T11:19:25.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cebfcd640 10 monclient(hunting): picked mon.noname-c con 0x7f4ce41406b0 addr v1:172.21.15.104:6790/0 2024-03-29T11:19:25.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cebfcd640 10 monclient(hunting): picked mon.noname-b con 0x7f4ce40b07d0 addr v1:172.21.15.119:6789/0 2024-03-29T11:19:25.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cebfcd640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ce4142220 con 0x7f4ce409ac40 2024-03-29T11:19:25.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cebfcd640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ce41399e0 con 0x7f4ce40b07d0 2024-03-29T11:19:25.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cebfcd640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ce4143990 con 0x7f4ce41406b0 2024-03-29T11:19:25.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cebfcd640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:25.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cebfcd640 10 monclient(hunting): authenticate will time out at 942.769210s 2024-03-29T11:19:25.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cea56a640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f4ce40b07d0 0x7f4ce41392d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:36608/0 (socket says 172.21.15.119:36608) 2024-03-29T11:19:25.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cea56a640 1 -- 172.21.15.119:0/963491235 learned_addr learned my addr 172.21.15.119:0/963491235 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-03-29T11:19:25.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 10 client.?.objecter ms_handle_connect 0x7f4ce40b07d0 2024-03-29T11:19:25.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:25.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 10 client.?.objecter ms_handle_connect 0x7f4ce409ac40 2024-03-29T11:19:25.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:25.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 1 -- 172.21.15.119:0/963491235 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2369104348 0 0) 0x7f4ce41399e0 con 0x7f4ce40b07d0 2024-03-29T11:19:25.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:25.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:25.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:25.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 10 monclient(hunting): my global_id is 4117 2024-03-29T11:19:25.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 1 -- 172.21.15.119:0/963491235 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cc8004eb0 con 0x7f4ce40b07d0 2024-03-29T11:19:25.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 10 client.?.objecter ms_handle_connect 0x7f4ce41406b0 2024-03-29T11:19:25.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:25.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 1 -- 172.21.15.119:0/963491235 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2762880421 0 0) 0x7f4ce4142220 con 0x7f4ce409ac40 2024-03-29T11:19:25.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:25.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:25.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:25.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 10 monclient(hunting): my global_id is 4131 2024-03-29T11:19:25.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 1 -- 172.21.15.119:0/963491235 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ce41399e0 con 0x7f4ce409ac40 2024-03-29T11:19:25.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 1 -- 172.21.15.119:0/963491235 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 23468112 0 0) 0x7f4ce4143990 con 0x7f4ce41406b0 2024-03-29T11:19:25.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:25.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:25.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:25.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 10 monclient(hunting): my global_id is 4127 2024-03-29T11:19:25.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 1 -- 172.21.15.119:0/963491235 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ce4142220 con 0x7f4ce41406b0 2024-03-29T11:19:25.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 1 -- 172.21.15.119:0/963491235 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2747919156 0 0) 0x7f4cc8004eb0 con 0x7f4ce40b07d0 2024-03-29T11:19:25.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 1 -- 172.21.15.119:0/963491235 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ce4143990 con 0x7f4ce40b07d0 2024-03-29T11:19:25.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 1 -- 172.21.15.119:0/963491235 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f4cdc001f40 con 0x7f4ce40b07d0 2024-03-29T11:19:25.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:25.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:25.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 10 monclient(hunting): dump: 2024-03-29T11:19:25.484 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 1 2024-03-29T11:19:25.484 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:25.484 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.484 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.484 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:25.484 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:25.485 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:25.485 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:25.485 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:25.485 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:25.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:25.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 1 -- 172.21.15.119:0/963491235 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3729579915 0 0) 0x7f4ce41399e0 con 0x7f4ce409ac40 2024-03-29T11:19:25.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 1 -- 172.21.15.119:0/963491235 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4cc8004eb0 con 0x7f4ce409ac40 2024-03-29T11:19:25.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 1 -- 172.21.15.119:0/963491235 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f4ce0001870 con 0x7f4ce409ac40 2024-03-29T11:19:25.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:25.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:25.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 10 monclient(hunting): dump: 2024-03-29T11:19:25.485 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 1 2024-03-29T11:19:25.485 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:25.485 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.485 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.485 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:25.485 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:25.486 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:25.486 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:25.486 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:25.486 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:25.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 1 -- 172.21.15.119:0/963491235 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 459246518 0 0) 0x7f4ce4142220 con 0x7f4ce41406b0 2024-03-29T11:19:25.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 1 -- 172.21.15.119:0/963491235 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ce41399e0 con 0x7f4ce41406b0 2024-03-29T11:19:25.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 1 -- 172.21.15.119:0/963491235 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f4cd40018a0 con 0x7f4ce41406b0 2024-03-29T11:19:25.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:25.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:25.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 10 monclient(hunting): dump: 2024-03-29T11:19:25.486 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 1 2024-03-29T11:19:25.486 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:25.486 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.486 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.487 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:25.487 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:25.487 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:25.487 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:25.487 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:25.487 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:25.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 1 -- 172.21.15.119:0/963491235 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3689842032 0 0) 0x7f4ce4143990 con 0x7f4ce40b07d0 2024-03-29T11:19:25.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 1 -- 172.21.15.119:0/963491235 >> v1:172.21.15.104:6790/0 conn(0x7f4ce41406b0 legacy=0x7f4ce4138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:25.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 1 -- 172.21.15.119:0/963491235 >> v1:172.21.15.104:6789/0 conn(0x7f4ce409ac40 legacy=0x7f4ce413ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:25.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:19:25.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 1 monclient: found mon.b 2024-03-29T11:19:25.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:25.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:19:25.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 1 -- 172.21.15.119:0/963491235 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce413a1b0 con 0x7f4ce40b07d0 2024-03-29T11:19:25.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 10 monclient: _finish_auth 0 2024-03-29T11:19:25.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.492+0000 7f4cda7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:19:25.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.493+0000 7f4cda7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:25.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.493+0000 7f4cebfcd640 5 monclient: authenticate success, global_id 4117 2024-03-29T11:19:25.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.493+0000 7f4cebfcd640 10 monclient: _renew_subs 2024-03-29T11:19:25.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.493+0000 7f4cebfcd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:19:25.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.493+0000 7f4cebfcd640 1 -- 172.21.15.119:0/963491235 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ce4141330 con 0x7f4ce40b07d0 2024-03-29T11:19:25.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.493+0000 7f4cebfcd640 10 client.4117.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:25.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.493+0000 7f4cebfcd640 10 monclient: _renew_subs 2024-03-29T11:19:25.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.493+0000 7f4cebfcd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:19:25.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.493+0000 7f4cebfcd640 1 -- 172.21.15.119:0/963491235 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ce41426b0 con 0x7f4ce40b07d0 2024-03-29T11:19:25.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.493+0000 7f4cebfcd640 1 librados: init done 2024-03-29T11:19:25.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.493+0000 7f4cda7fc640 1 -- 172.21.15.119:0/963491235 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4cdc004e10 con 0x7f4ce40b07d0 2024-03-29T11:19:25.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.493+0000 7f4cda7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:25.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.493+0000 7f4cdbfff640 4 set_mon_vals no callback set 2024-03-29T11:19:25.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.493+0000 7f4cda7fc640 1 -- 172.21.15.119:0/963491235 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f4cdc005120 con 0x7f4ce40b07d0 2024-03-29T11:19:25.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.493+0000 7f4cda7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:25.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.493+0000 7f4cda7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:25.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.493+0000 7f4cda7fc640 10 monclient: dump: 2024-03-29T11:19:25.489 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 1 2024-03-29T11:19:25.489 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:25.489 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.489 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.489 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:25.489 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:25.489 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:25.489 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:25.489 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:25.489 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:25.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.493+0000 7f4cda7fc640 1 -- 172.21.15.119:0/963491235 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f4cdc005610 con 0x7f4ce40b07d0 2024-03-29T11:19:25.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.493+0000 7f4cda7fc640 1 -- 172.21.15.119:0/963491235 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 2467120252 0 0) 0x7f4cdc006bb0 con 0x7f4ce40b07d0 2024-03-29T11:19:25.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.493+0000 7f4cda7fc640 10 client.4117.objecter ms_dispatch 0x7f4ce4000b90 osd_map(4..4 src has 1..4) v4 2024-03-29T11:19:25.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.493+0000 7f4cda7fc640 3 client.4117.objecter handle_osd_map got epochs [4,4] > 0 2024-03-29T11:19:25.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.493+0000 7f4cda7fc640 3 client.4117.objecter handle_osd_map decoding full epoch 4 2024-03-29T11:19:25.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.493+0000 7f4cda7fc640 20 client.4117.objecter dump_active .. 0 homeless 2024-03-29T11:19:25.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.493+0000 7f4cebfcd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:25.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.493+0000 7f4cebfcd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:25.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.493+0000 7f4cebfcd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:19:25.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.493+0000 7f4cebfcd640 1 -- 172.21.15.119:0/963491235 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ccc005080 con 0x7f4ce40b07d0 2024-03-29T11:19:25.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.497+0000 7f4cda7fc640 1 -- 172.21.15.119:0/963491235 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f4cdc006780 con 0x7f4ce40b07d0 2024-03-29T11:19:25.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.497+0000 7f4cda7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:25.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.497+0000 7f4cda7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:25.490 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-03-29T11:19:25.503+0000 7f32f3443040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:19:25.525 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-29T11:19:25.548+0000 7f0ede043040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:19:25.535 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-03-29T11:19:25.558+0000 7f32f3443040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:19:25.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.570+0000 7f4cebfcd640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "9bcd508f-a564-47de-bc8d-d287c8a0441f", "id": 2}] 2024-03-29T11:19:25.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.570+0000 7f4cebfcd640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "9bcd508f-a564-47de-bc8d-d287c8a0441f", "id": 2}] 2024-03-29T11:19:25.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.570+0000 7f4cebfcd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:19:25.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.570+0000 7f4cebfcd640 1 -- 172.21.15.119:0/963491235 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "osd new", "uuid": "9bcd508f-a564-47de-bc8d-d287c8a0441f", "id": 2} v 0) v1 -- 0x7f4ccc0058b0 con 0x7f4ce40b07d0 2024-03-29T11:19:25.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.578+0000 7f4cda7fc640 1 -- 172.21.15.119:0/963491235 <== mon.1 v1:172.21.15.119:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9bcd508f-a564-47de-bc8d-d287c8a0441f", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 3664065740 0 216599125) 0x7f4cdc005aa0 con 0x7f4ce40b07d0 2024-03-29T11:19:25.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.578+0000 7f4cda7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "9bcd508f-a564-47de-bc8d-d287c8a0441f", "id": 2}] 2024-03-29T11:19:25.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.578+0000 7f4cda7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:25.555 INFO:teuthology.orchestra.run.smithi119.stdout:2 2024-03-29T11:19:25.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.579+0000 7f4cebfcd640 10 librados: watch_flush enter 2024-03-29T11:19:25.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.579+0000 7f4cebfcd640 10 librados: watch_flush exit 2024-03-29T11:19:25.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.579+0000 7f4cebfcd640 20 client.4117.objecter shutdown clearing up homeless session... 2024-03-29T11:19:25.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.579+0000 7f4cebfcd640 10 client.4117.objecter successfully canceled tick 2024-03-29T11:19:25.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.579+0000 7f4cebfcd640 10 monclient: shutdown 2024-03-29T11:19:25.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.579+0000 7f4cebfcd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:25.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.579+0000 7f4cebfcd640 1 -- 172.21.15.119:0/963491235 >> v1:172.21.15.119:6789/0 conn(0x7f4ce40b07d0 legacy=0x7f4ce41392d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:25.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.579+0000 7f4cebfcd640 1 -- 172.21.15.119:0/963491235 shutdown_connections 2024-03-29T11:19:25.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.579+0000 7f4cebfcd640 1 -- 172.21.15.119:0/963491235 >> 172.21.15.119:0/963491235 conn(0x7f4ce4066b00 msgr2=0x7f4ce413b2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:25.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.579+0000 7f4cebfcd640 1 -- 172.21.15.119:0/963491235 shutdown_connections 2024-03-29T11:19:25.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.579+0000 7f4cebfcd640 1 -- 172.21.15.119:0/963491235 wait complete. 2024-03-29T11:19:25.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.579+0000 7f4cebfcd640 1 librados: shutdown 2024-03-29T11:19:25.566 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster ceph osd new 991d3031-9687-404a-a087-a3781ccfdc63 3 2024-03-29T11:19:25.571 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-29T11:19:25.594+0000 7f0ede043040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:19:25.583 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-03-29T11:19:25.607+0000 7f32f3443040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:19:25.624 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-29T11:19:25.647+0000 7f0ede043040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:19:25.640 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-03-29T11:19:25.663+0000 7f32f3443040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:19:25.665 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-29T11:19:25.689+0000 7f0ede043040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:19:25.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.689+0000 7f057530e640 10 monclient: get_monmap_and_config 2024-03-29T11:19:25.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.689+0000 7f057530e640 10 monclient: build_initial_monmap 2024-03-29T11:19:25.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.689+0000 7f057530e640 10 monclient: monmap: 2024-03-29T11:19:25.666 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 0 2024-03-29T11:19:25.666 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:25.666 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:19:25.690543+0000 2024-03-29T11:19:25.666 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:19:25.690543+0000 2024-03-29T11:19:25.666 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:25.666 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:25.667 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:19:25.667 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:19:25.667 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:19:25.667 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:25.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.689+0000 7f057530e640 1 Processor -- start 2024-03-29T11:19:25.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.689+0000 7f057530e640 1 -- start start 2024-03-29T11:19:25.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.689+0000 7f057530e640 10 monclient: init 2024-03-29T11:19:25.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.690+0000 7f057530e640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:25.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.690+0000 7f057530e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:25.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.690+0000 7f057530e640 10 monclient(hunting): picked mon.noname-c con 0x7f05700b8240 addr v1:172.21.15.119:6789/0 2024-03-29T11:19:25.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.690+0000 7f057530e640 10 monclient(hunting): picked mon.noname-a con 0x7f05700b1f60 addr v1:172.21.15.104:6789/0 2024-03-29T11:19:25.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.690+0000 7f057530e640 10 monclient(hunting): picked mon.noname-b con 0x7f057009c8c0 addr v1:172.21.15.104:6790/0 2024-03-29T11:19:25.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.690+0000 7f057530e640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f057009bd80 con 0x7f05700b1f60 2024-03-29T11:19:25.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.690+0000 7f057530e640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f057009b960 con 0x7f05700b8240 2024-03-29T11:19:25.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.690+0000 7f057530e640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05700b2ec0 con 0x7f057009c8c0 2024-03-29T11:19:25.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.690+0000 7f057530e640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:25.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.690+0000 7f057530e640 10 monclient(hunting): authenticate will time out at 942.967644s 2024-03-29T11:19:25.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.690+0000 7f056effd640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f05700b8240 0x7f057013f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:48396/0 (socket says 172.21.15.119:48396) 2024-03-29T11:19:25.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.690+0000 7f056effd640 1 -- 172.21.15.119:0/1632714466 learned_addr learned my addr 172.21.15.119:0/1632714466 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-03-29T11:19:25.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.690+0000 7f056dffb640 1 -- 172.21.15.119:0/1632714466 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 322740532 0 0) 0x7f057009b960 con 0x7f05700b8240 2024-03-29T11:19:25.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.690+0000 7f056dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:25.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.690+0000 7f056dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:25.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.690+0000 7f056dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:25.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.690+0000 7f056dffb640 10 monclient(hunting): my global_id is 4120 2024-03-29T11:19:25.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.690+0000 7f056dffb640 1 -- 172.21.15.119:0/1632714466 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0558003520 con 0x7f05700b8240 2024-03-29T11:19:25.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.691+0000 7f056dffb640 1 -- 172.21.15.119:0/1632714466 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1431358052 0 0) 0x7f057009bd80 con 0x7f05700b1f60 2024-03-29T11:19:25.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.691+0000 7f056dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:25.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.691+0000 7f056dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:25.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.691+0000 7f056dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:25.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.691+0000 7f056dffb640 10 monclient(hunting): my global_id is 4134 2024-03-29T11:19:25.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.691+0000 7f056dffb640 1 -- 172.21.15.119:0/1632714466 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f057009b960 con 0x7f05700b1f60 2024-03-29T11:19:25.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.691+0000 7f056dffb640 1 -- 172.21.15.119:0/1632714466 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3893769044 0 0) 0x7f05700b2ec0 con 0x7f057009c8c0 2024-03-29T11:19:25.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.691+0000 7f056dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:25.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.691+0000 7f056dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:25.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.691+0000 7f056dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:25.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.691+0000 7f056dffb640 10 monclient(hunting): my global_id is 4130 2024-03-29T11:19:25.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.691+0000 7f056dffb640 1 -- 172.21.15.119:0/1632714466 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f057009bd80 con 0x7f057009c8c0 2024-03-29T11:19:25.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.691+0000 7f056dffb640 1 -- 172.21.15.119:0/1632714466 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2614355937 0 0) 0x7f0558003520 con 0x7f05700b8240 2024-03-29T11:19:25.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.691+0000 7f056dffb640 1 -- 172.21.15.119:0/1632714466 >> v1:172.21.15.104:6790/0 conn(0x7f057009c8c0 legacy=0x7f057014d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:25.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.691+0000 7f056dffb640 1 -- 172.21.15.119:0/1632714466 >> v1:172.21.15.104:6789/0 conn(0x7f05700b1f60 legacy=0x7f057009c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:25.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.691+0000 7f056dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:19:25.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.691+0000 7f056dffb640 1 monclient: found mon.noname-c 2024-03-29T11:19:25.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.691+0000 7f056dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:25.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.691+0000 7f056dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.119:6789/0 2024-03-29T11:19:25.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.691+0000 7f056dffb640 1 -- 172.21.15.119:0/1632714466 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f057013fb60 con 0x7f05700b8240 2024-03-29T11:19:25.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.691+0000 7f056dffb640 10 monclient: _finish_auth 0 2024-03-29T11:19:25.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.691+0000 7f056dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:19:25.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.691+0000 7f056dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:25.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.691+0000 7f056dffb640 1 -- 172.21.15.119:0/1632714466 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f0564001690 con 0x7f05700b8240 2024-03-29T11:19:25.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.691+0000 7f057530e640 5 monclient: authenticate success, global_id 4120 2024-03-29T11:19:25.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.691+0000 7f057530e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:25.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.691+0000 7f056dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:25.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.691+0000 7f056dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:25.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.691+0000 7f056dffb640 10 monclient: dump: 2024-03-29T11:19:25.679 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 1 2024-03-29T11:19:25.679 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:25.679 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.679 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.679 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:25.679 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:25.679 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:25.679 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:25.679 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:25.679 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:25.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.691+0000 7f057530e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:25.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.691+0000 7f056dffb640 1 -- 172.21.15.119:0/1632714466 <== mon.1 v1:172.21.15.119:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f05640017f0 con 0x7f05700b8240 2024-03-29T11:19:25.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.691+0000 7f056dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:25.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.691+0000 7f057530e640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:25.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.691+0000 7f056dffb640 1 -- 172.21.15.119:0/1632714466 <== mon.1 v1:172.21.15.119:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f0564001c00 con 0x7f05700b8240 2024-03-29T11:19:25.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.691+0000 7f057530e640 4 set_mon_vals no callback set 2024-03-29T11:19:25.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.691+0000 7f056dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:25.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.691+0000 7f056dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:25.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.691+0000 7f056dffb640 10 monclient: dump: 2024-03-29T11:19:25.680 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 1 2024-03-29T11:19:25.680 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:25.680 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.680 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.680 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:25.680 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:25.680 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:25.680 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:25.680 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:25.680 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:25.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.691+0000 7f057530e640 10 monclient: shutdown 2024-03-29T11:19:25.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.691+0000 7f057530e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:25.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.691+0000 7f057530e640 1 -- 172.21.15.119:0/1632714466 >> v1:172.21.15.119:6789/0 conn(0x7f05700b8240 legacy=0x7f057013f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:25.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.691+0000 7f057530e640 1 -- 172.21.15.119:0/1632714466 shutdown_connections 2024-03-29T11:19:25.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.691+0000 7f057530e640 1 -- 172.21.15.119:0/1632714466 >> 172.21.15.119:0/1632714466 conn(0x7f0570066b00 msgr2=0x7f0570066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:25.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.692+0000 7f057530e640 1 -- 172.21.15.119:0/1632714466 shutdown_connections 2024-03-29T11:19:25.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.692+0000 7f057530e640 1 -- 172.21.15.119:0/1632714466 wait complete. 2024-03-29T11:19:25.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.692+0000 7f057530e640 10 monclient: build_initial_monmap 2024-03-29T11:19:25.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.692+0000 7f057530e640 10 monclient: monmap: 2024-03-29T11:19:25.681 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 0 2024-03-29T11:19:25.681 INFO:teuthology.orchestra.run.smithi119.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:25.681 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 0.000000 2024-03-29T11:19:25.681 INFO:teuthology.orchestra.run.smithi119.stderr:created 0.000000 2024-03-29T11:19:25.681 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:25.681 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:25.681 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:19:25.681 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:19:25.681 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:19:25.682 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:25.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.692+0000 7f057530e640 1 librados: starting msgr at 2024-03-29T11:19:25.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.692+0000 7f057530e640 1 librados: starting objecter 2024-03-29T11:19:25.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.693+0000 7f057530e640 1 Processor -- start 2024-03-29T11:19:25.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.693+0000 7f057530e640 1 -- start start 2024-03-29T11:19:25.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.693+0000 7f057530e640 1 librados: setting wanted keys 2024-03-29T11:19:25.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.693+0000 7f057530e640 1 librados: calling monclient init 2024-03-29T11:19:25.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.693+0000 7f057530e640 10 monclient: init 2024-03-29T11:19:25.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.693+0000 7f057530e640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:25.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.693+0000 7f057530e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:25.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.693+0000 7f057530e640 10 monclient(hunting): picked mon.noname-a con 0x7f057009c8c0 addr v1:172.21.15.104:6789/0 2024-03-29T11:19:25.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.693+0000 7f057530e640 10 monclient(hunting): picked mon.noname-c con 0x7f05700b8240 addr v1:172.21.15.104:6790/0 2024-03-29T11:19:25.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.693+0000 7f057530e640 10 monclient(hunting): picked mon.noname-b con 0x7f05700b0510 addr v1:172.21.15.119:6789/0 2024-03-29T11:19:25.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.693+0000 7f057530e640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05700b0960 con 0x7f057009c8c0 2024-03-29T11:19:25.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.693+0000 7f057530e640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05700b0160 con 0x7f05700b0510 2024-03-29T11:19:25.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.693+0000 7f057530e640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05701da8e0 con 0x7f05700b8240 2024-03-29T11:19:25.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.693+0000 7f057530e640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:25.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.693+0000 7f057530e640 10 monclient(hunting): authenticate will time out at 942.970686s 2024-03-29T11:19:25.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.693+0000 7f056effd640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f057009c8c0 0x7f05700adc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.119:42652/0 (socket says 172.21.15.119:42652) 2024-03-29T11:19:25.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.693+0000 7f056effd640 1 -- 172.21.15.119:0/1815019612 learned_addr learned my addr 172.21.15.119:0/1815019612 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-03-29T11:19:25.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.693+0000 7f056f7fe640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f05700b0510 0x7f0570134d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:48400/0 (socket says 172.21.15.119:48400) 2024-03-29T11:19:25.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.693+0000 7f05577fe640 10 client.?.objecter ms_handle_connect 0x7f05700b0510 2024-03-29T11:19:25.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.693+0000 7f05577fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:25.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.693+0000 7f05577fe640 10 client.?.objecter ms_handle_connect 0x7f057009c8c0 2024-03-29T11:19:25.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.693+0000 7f05577fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:25.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.693+0000 7f05577fe640 10 client.?.objecter ms_handle_connect 0x7f05700b8240 2024-03-29T11:19:25.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.693+0000 7f05577fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:25.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.693+0000 7f05577fe640 1 -- 172.21.15.119:0/1815019612 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2345333742 0 0) 0x7f05700b0160 con 0x7f05700b0510 2024-03-29T11:19:25.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.693+0000 7f05577fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:25.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.693+0000 7f05577fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:25.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.693+0000 7f05577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:25.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.693+0000 7f05577fe640 10 monclient(hunting): my global_id is 4123 2024-03-29T11:19:25.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.693+0000 7f05577fe640 1 -- 172.21.15.119:0/1815019612 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f055c004eb0 con 0x7f05700b0510 2024-03-29T11:19:25.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 1 -- 172.21.15.119:0/1815019612 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1777814455 0 0) 0x7f05700b0960 con 0x7f057009c8c0 2024-03-29T11:19:25.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:25.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:25.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:25.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 10 monclient(hunting): my global_id is 4137 2024-03-29T11:19:25.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 1 -- 172.21.15.119:0/1815019612 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05700b0160 con 0x7f057009c8c0 2024-03-29T11:19:25.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 1 -- 172.21.15.119:0/1815019612 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 909797810 0 0) 0x7f05701da8e0 con 0x7f05700b8240 2024-03-29T11:19:25.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:25.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:25.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:25.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 10 monclient(hunting): my global_id is 4133 2024-03-29T11:19:25.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 1 -- 172.21.15.119:0/1815019612 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05700b0960 con 0x7f05700b8240 2024-03-29T11:19:25.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 1 -- 172.21.15.119:0/1815019612 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1991750604 0 0) 0x7f05700b0160 con 0x7f057009c8c0 2024-03-29T11:19:25.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 1 -- 172.21.15.119:0/1815019612 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f05701da8e0 con 0x7f057009c8c0 2024-03-29T11:19:25.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 1 -- 172.21.15.119:0/1815019612 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f0564001b40 con 0x7f057009c8c0 2024-03-29T11:19:25.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:25.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:25.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 10 monclient(hunting): dump: 2024-03-29T11:19:25.685 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 1 2024-03-29T11:19:25.685 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:25.685 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.685 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.685 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:25.685 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:25.685 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:25.686 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:25.686 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:25.686 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:25.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:25.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 1 -- 172.21.15.119:0/1815019612 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2108772990 0 0) 0x7f055c004eb0 con 0x7f05700b0510 2024-03-29T11:19:25.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 1 -- 172.21.15.119:0/1815019612 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f05700b0160 con 0x7f05700b0510 2024-03-29T11:19:25.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 1 -- 172.21.15.119:0/1815019612 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 988955782 0 0) 0x7f05700b0960 con 0x7f05700b8240 2024-03-29T11:19:25.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 1 -- 172.21.15.119:0/1815019612 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f055c004eb0 con 0x7f05700b8240 2024-03-29T11:19:25.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 1 -- 172.21.15.119:0/1815019612 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f0568001bf0 con 0x7f05700b0510 2024-03-29T11:19:25.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:25.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:25.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 10 monclient(hunting): dump: 2024-03-29T11:19:25.686 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 1 2024-03-29T11:19:25.686 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:25.687 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.687 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.687 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:25.687 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:25.687 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:25.687 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:25.687 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:25.687 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:25.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 1 -- 172.21.15.119:0/1815019612 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f05600018b0 con 0x7f05700b8240 2024-03-29T11:19:25.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:25.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:25.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 10 monclient(hunting): dump: 2024-03-29T11:19:25.687 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 1 2024-03-29T11:19:25.687 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:25.687 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.688 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.688 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:25.688 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:25.688 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:25.688 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:25.688 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:25.688 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:25.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 1 -- 172.21.15.119:0/1815019612 <== mon.0 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2147616418 0 0) 0x7f05701da8e0 con 0x7f057009c8c0 2024-03-29T11:19:25.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 1 -- 172.21.15.119:0/1815019612 >> v1:172.21.15.104:6790/0 conn(0x7f05700b8240 legacy=0x7f05701dae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:25.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 1 -- 172.21.15.119:0/1815019612 >> v1:172.21.15.119:6789/0 conn(0x7f05700b0510 legacy=0x7f0570134d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:25.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:25.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 1 monclient: found mon.a 2024-03-29T11:19:25.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:25.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:19:25.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 1 -- 172.21.15.119:0/1815019612 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05701db510 con 0x7f057009c8c0 2024-03-29T11:19:25.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 10 monclient: _finish_auth 0 2024-03-29T11:19:25.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:25.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:25.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f057530e640 5 monclient: authenticate success, global_id 4137 2024-03-29T11:19:25.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f057530e640 10 monclient: _renew_subs 2024-03-29T11:19:25.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f057530e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:19:25.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f057530e640 1 -- 172.21.15.119:0/1815019612 --> v1:172.21.15.104:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0570135470 con 0x7f057009c8c0 2024-03-29T11:19:25.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f057530e640 10 client.4137.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:25.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f057530e640 10 monclient: _renew_subs 2024-03-29T11:19:25.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f057530e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:19:25.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f057530e640 1 -- 172.21.15.119:0/1815019612 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f05700aeab0 con 0x7f057009c8c0 2024-03-29T11:19:25.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f057530e640 1 librados: init done 2024-03-29T11:19:25.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 1 -- 172.21.15.119:0/1815019612 <== mon.0 v1:172.21.15.104:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0564004a90 con 0x7f057009c8c0 2024-03-29T11:19:25.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:25.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 1 -- 172.21.15.119:0/1815019612 <== mon.0 v1:172.21.15.104:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f0564004da0 con 0x7f057009c8c0 2024-03-29T11:19:25.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:25.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:25.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f056d7fa640 4 set_mon_vals no callback set 2024-03-29T11:19:25.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 10 monclient: dump: 2024-03-29T11:19:25.690 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 1 2024-03-29T11:19:25.690 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:25.690 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.690 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.690 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:25.690 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:25.690 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:25.690 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:25.690 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:25.690 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:25.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 1 -- 172.21.15.119:0/1815019612 <== mon.0 v1:172.21.15.104:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f0564005290 con 0x7f057009c8c0 2024-03-29T11:19:25.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 1 -- 172.21.15.119:0/1815019612 <== mon.0 v1:172.21.15.104:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 135455374 0 0) 0x7f0564006810 con 0x7f057009c8c0 2024-03-29T11:19:25.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 10 client.4137.objecter ms_dispatch 0x7f0570000b90 osd_map(5..5 src has 1..5) v4 2024-03-29T11:19:25.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 3 client.4137.objecter handle_osd_map got epochs [5,5] > 0 2024-03-29T11:19:25.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 3 client.4137.objecter handle_osd_map decoding full epoch 5 2024-03-29T11:19:25.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.694+0000 7f05577fe640 20 client.4137.objecter dump_active .. 0 homeless 2024-03-29T11:19:25.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.695+0000 7f057530e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:25.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.695+0000 7f057530e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:25.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.695+0000 7f057530e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:19:25.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.695+0000 7f057530e640 1 -- 172.21.15.119:0/1815019612 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0550005080 con 0x7f057009c8c0 2024-03-29T11:19:25.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.697+0000 7f05577fe640 1 -- 172.21.15.119:0/1815019612 <== mon.0 v1:172.21.15.104:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f0564006380 con 0x7f057009c8c0 2024-03-29T11:19:25.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.697+0000 7f05577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:25.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.697+0000 7f05577fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:25.692 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-03-29T11:19:25.709+0000 7f32f3443040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:19:25.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.769+0000 7f057530e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "991d3031-9687-404a-a087-a3781ccfdc63", "id": 3}] 2024-03-29T11:19:25.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.769+0000 7f057530e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "991d3031-9687-404a-a087-a3781ccfdc63", "id": 3}] 2024-03-29T11:19:25.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.769+0000 7f057530e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:19:25.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.769+0000 7f057530e640 1 -- 172.21.15.119:0/1815019612 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "osd new", "uuid": "991d3031-9687-404a-a087-a3781ccfdc63", "id": 3} v 0) v1 -- 0x7f05500058b0 con 0x7f057009c8c0 2024-03-29T11:19:25.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.771+0000 7f05577fe640 1 -- 172.21.15.119:0/1815019612 <== mon.0 v1:172.21.15.104:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "991d3031-9687-404a-a087-a3781ccfdc63", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 1672032539 0 525046306) 0x7f0564006aa0 con 0x7f057009c8c0 2024-03-29T11:19:25.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.771+0000 7f05577fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "991d3031-9687-404a-a087-a3781ccfdc63", "id": 3}] 2024-03-29T11:19:25.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.771+0000 7f05577fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:25.748 INFO:teuthology.orchestra.run.smithi119.stdout:3 2024-03-29T11:19:25.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.772+0000 7f057530e640 10 librados: watch_flush enter 2024-03-29T11:19:25.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.772+0000 7f057530e640 10 librados: watch_flush exit 2024-03-29T11:19:25.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.772+0000 7f057530e640 20 client.4137.objecter shutdown clearing up homeless session... 2024-03-29T11:19:25.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.772+0000 7f057530e640 10 client.4137.objecter successfully canceled tick 2024-03-29T11:19:25.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.772+0000 7f057530e640 10 monclient: shutdown 2024-03-29T11:19:25.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.772+0000 7f057530e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:25.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.772+0000 7f057530e640 1 -- 172.21.15.119:0/1815019612 >> v1:172.21.15.104:6789/0 conn(0x7f057009c8c0 legacy=0x7f05700adc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:25.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.772+0000 7f057530e640 1 -- 172.21.15.119:0/1815019612 shutdown_connections 2024-03-29T11:19:25.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.772+0000 7f057530e640 1 -- 172.21.15.119:0/1815019612 >> 172.21.15.119:0/1815019612 conn(0x7f0570066b00 msgr2=0x7f05700b70e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:25.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.772+0000 7f057530e640 1 -- 172.21.15.119:0/1815019612 shutdown_connections 2024-03-29T11:19:25.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.772+0000 7f057530e640 1 -- 172.21.15.119:0/1815019612 wait complete. 2024-03-29T11:19:25.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.772+0000 7f057530e640 1 librados: shutdown 2024-03-29T11:19:25.759 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster ceph osd new 010f8945-19c0-4f55-b2fc-21d83b076f4e 4 2024-03-29T11:19:25.782 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-29T11:19:25.805+0000 7f0ede043040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:19:25.806 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-03-29T11:19:25.829+0000 7f32f3443040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:19:25.850 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-29T11:19:25.873+0000 7f0ede043040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:19:25.880 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-03-29T11:19:25.903+0000 7f32f3443040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:19:25.895 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-29T11:19:25.918+0000 7f0ede043040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:19:25.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.923+0000 7f724f425640 10 monclient: get_monmap_and_config 2024-03-29T11:19:25.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.923+0000 7f724f425640 10 monclient: build_initial_monmap 2024-03-29T11:19:25.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.923+0000 7f724f425640 10 monclient: monmap: 2024-03-29T11:19:25.900 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 0 2024-03-29T11:19:25.900 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:25.900 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:19:25.924854+0000 2024-03-29T11:19:25.900 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:19:25.924854+0000 2024-03-29T11:19:25.901 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:25.901 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:25.901 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:19:25.901 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:19:25.901 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:19:25.901 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:25.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.924+0000 7f724f425640 1 Processor -- start 2024-03-29T11:19:25.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.924+0000 7f724f425640 1 -- start start 2024-03-29T11:19:25.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.924+0000 7f724f425640 10 monclient: init 2024-03-29T11:19:25.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.924+0000 7f724f425640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:25.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.924+0000 7f724f425640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:25.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.924+0000 7f724f425640 10 monclient(hunting): picked mon.noname-a con 0x7f724809ac40 addr v1:172.21.15.104:6789/0 2024-03-29T11:19:25.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.924+0000 7f724f425640 10 monclient(hunting): picked mon.noname-b con 0x7f724809c870 addr v1:172.21.15.104:6790/0 2024-03-29T11:19:25.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.924+0000 7f724f425640 10 monclient(hunting): picked mon.noname-c con 0x7f72480b4360 addr v1:172.21.15.119:6789/0 2024-03-29T11:19:25.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.924+0000 7f724f425640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f724809dec0 con 0x7f724809ac40 2024-03-29T11:19:25.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f724f425640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f724809da30 con 0x7f72480b4360 2024-03-29T11:19:25.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f724f425640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72480b39c0 con 0x7f724809c870 2024-03-29T11:19:25.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f724f425640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:25.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f724f425640 10 monclient(hunting): authenticate will time out at 943.202054s 2024-03-29T11:19:25.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f724d9c2640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f72480b4360 0x7f724814d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:48402/0 (socket says 172.21.15.119:48402) 2024-03-29T11:19:25.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f724d9c2640 1 -- 172.21.15.119:0/1034746029 learned_addr learned my addr 172.21.15.119:0/1034746029 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-03-29T11:19:25.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f723ffff640 1 -- 172.21.15.119:0/1034746029 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 153104359 0 0) 0x7f724809da30 con 0x7f72480b4360 2024-03-29T11:19:25.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f723ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:25.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f723ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:25.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f723ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:25.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f723ffff640 10 monclient(hunting): my global_id is 4126 2024-03-29T11:19:25.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f723ffff640 1 -- 172.21.15.119:0/1034746029 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7234003520 con 0x7f72480b4360 2024-03-29T11:19:25.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f723ffff640 1 -- 172.21.15.119:0/1034746029 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 605206402 0 0) 0x7f7244001110 con 0x7f724809ac40 2024-03-29T11:19:25.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f723ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:25.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f723ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:25.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f723ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:25.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f723ffff640 10 monclient(hunting): my global_id is 4140 2024-03-29T11:19:25.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f723ffff640 1 -- 172.21.15.119:0/1034746029 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f724809da30 con 0x7f724809ac40 2024-03-29T11:19:25.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f723ffff640 1 -- 172.21.15.119:0/1034746029 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 717942659 0 0) 0x7f72480b39c0 con 0x7f724809c870 2024-03-29T11:19:25.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f723ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:25.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f723ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:25.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f723ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:25.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f723ffff640 10 monclient(hunting): my global_id is 4136 2024-03-29T11:19:25.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f723ffff640 1 -- 172.21.15.119:0/1034746029 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7244001110 con 0x7f724809c870 2024-03-29T11:19:25.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f723ffff640 1 -- 172.21.15.119:0/1034746029 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3008341700 0 0) 0x7f7234003520 con 0x7f72480b4360 2024-03-29T11:19:25.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f723ffff640 1 -- 172.21.15.119:0/1034746029 >> v1:172.21.15.104:6790/0 conn(0x7f724809c870 legacy=0x7f724813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:25.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f723ffff640 1 -- 172.21.15.119:0/1034746029 >> v1:172.21.15.104:6789/0 conn(0x7f724809ac40 legacy=0x7f72480b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:25.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f723ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:19:25.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f723ffff640 1 monclient: found mon.noname-c 2024-03-29T11:19:25.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f723ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:25.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f723ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.119:6789/0 2024-03-29T11:19:25.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f723ffff640 1 -- 172.21.15.119:0/1034746029 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72480b5160 con 0x7f72480b4360 2024-03-29T11:19:25.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f723ffff640 10 monclient: _finish_auth 0 2024-03-29T11:19:25.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f723ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:19:25.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f723ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:25.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f724f425640 5 monclient: authenticate success, global_id 4126 2024-03-29T11:19:25.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f723ffff640 1 -- 172.21.15.119:0/1034746029 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f7240001a10 con 0x7f72480b4360 2024-03-29T11:19:25.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f724f425640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:25.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f723ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:25.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f723ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:25.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f723ffff640 10 monclient: dump: 2024-03-29T11:19:25.914 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 1 2024-03-29T11:19:25.914 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:25.914 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.914 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.914 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:25.914 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:25.914 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:25.914 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:25.914 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:25.914 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:25.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f724f425640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:25.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f723ffff640 1 -- 172.21.15.119:0/1034746029 <== mon.1 v1:172.21.15.119:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7240001bf0 con 0x7f72480b4360 2024-03-29T11:19:25.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f723ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:25.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f723ffff640 1 -- 172.21.15.119:0/1034746029 <== mon.1 v1:172.21.15.119:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f7240002000 con 0x7f72480b4360 2024-03-29T11:19:25.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f724f425640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:25.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f724f425640 4 set_mon_vals no callback set 2024-03-29T11:19:25.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f723ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:25.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f723ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:25.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.925+0000 7f723ffff640 10 monclient: dump: 2024-03-29T11:19:25.915 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 1 2024-03-29T11:19:25.915 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:25.915 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.915 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.915 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:25.915 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:25.915 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:25.915 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:25.915 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:25.915 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:25.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.926+0000 7f724f425640 10 monclient: shutdown 2024-03-29T11:19:25.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.926+0000 7f724f425640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:25.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.926+0000 7f724f425640 1 -- 172.21.15.119:0/1034746029 >> v1:172.21.15.119:6789/0 conn(0x7f72480b4360 legacy=0x7f724814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:25.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.926+0000 7f724f425640 1 -- 172.21.15.119:0/1034746029 shutdown_connections 2024-03-29T11:19:25.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.926+0000 7f724f425640 1 -- 172.21.15.119:0/1034746029 >> 172.21.15.119:0/1034746029 conn(0x7f7248066b00 msgr2=0x7f7248066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:25.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.926+0000 7f724f425640 1 -- 172.21.15.119:0/1034746029 shutdown_connections 2024-03-29T11:19:25.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.926+0000 7f724f425640 1 -- 172.21.15.119:0/1034746029 wait complete. 2024-03-29T11:19:25.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.927+0000 7f724f425640 10 monclient: build_initial_monmap 2024-03-29T11:19:25.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.927+0000 7f724f425640 10 monclient: monmap: 2024-03-29T11:19:25.916 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 0 2024-03-29T11:19:25.916 INFO:teuthology.orchestra.run.smithi119.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:25.916 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 0.000000 2024-03-29T11:19:25.916 INFO:teuthology.orchestra.run.smithi119.stderr:created 0.000000 2024-03-29T11:19:25.916 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:25.916 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:25.916 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:19:25.917 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:19:25.917 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:19:25.917 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:25.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.927+0000 7f724f425640 1 librados: starting msgr at 2024-03-29T11:19:25.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.927+0000 7f724f425640 1 librados: starting objecter 2024-03-29T11:19:25.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.927+0000 7f724f425640 1 Processor -- start 2024-03-29T11:19:25.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.927+0000 7f724f425640 1 -- start start 2024-03-29T11:19:25.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.927+0000 7f724f425640 1 librados: setting wanted keys 2024-03-29T11:19:25.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.927+0000 7f724f425640 1 librados: calling monclient init 2024-03-29T11:19:25.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.927+0000 7f724f425640 10 monclient: init 2024-03-29T11:19:25.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.927+0000 7f724f425640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:25.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.927+0000 7f724f425640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:25.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f724f425640 10 monclient(hunting): picked mon.noname-c con 0x7f724809ac40 addr v1:172.21.15.104:6790/0 2024-03-29T11:19:25.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f724f425640 10 monclient(hunting): picked mon.noname-b con 0x7f72480b63e0 addr v1:172.21.15.119:6789/0 2024-03-29T11:19:25.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f724f425640 10 monclient(hunting): picked mon.noname-a con 0x7f72481461c0 addr v1:172.21.15.104:6789/0 2024-03-29T11:19:25.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f724f425640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72480b7520 con 0x7f72481461c0 2024-03-29T11:19:25.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f724f425640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72480b8960 con 0x7f72480b63e0 2024-03-29T11:19:25.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f724f425640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72480b7d30 con 0x7f724809ac40 2024-03-29T11:19:25.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f724f425640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:25.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f724f425640 10 monclient(hunting): authenticate will time out at 943.205133s 2024-03-29T11:19:25.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f724c9c0640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f72480b63e0 0x7f72481d4530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:48404/0 (socket says 172.21.15.119:48404) 2024-03-29T11:19:25.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f724c9c0640 1 -- 172.21.15.119:0/164331844 learned_addr learned my addr 172.21.15.119:0/164331844 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-03-29T11:19:25.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 10 client.?.objecter ms_handle_connect 0x7f72480b63e0 2024-03-29T11:19:25.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:25.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 10 client.?.objecter ms_handle_connect 0x7f724809ac40 2024-03-29T11:19:25.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:25.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 10 client.?.objecter ms_handle_connect 0x7f72481461c0 2024-03-29T11:19:25.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:25.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 1 -- 172.21.15.119:0/164331844 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1171558010 0 0) 0x7f72480b8960 con 0x7f72480b63e0 2024-03-29T11:19:25.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:25.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:25.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:25.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 10 monclient(hunting): my global_id is 4129 2024-03-29T11:19:25.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 1 -- 172.21.15.119:0/164331844 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f722c004eb0 con 0x7f72480b63e0 2024-03-29T11:19:25.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 1 -- 172.21.15.119:0/164331844 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1402484932 0 0) 0x7f72480b7d30 con 0x7f724809ac40 2024-03-29T11:19:25.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:25.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:25.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:25.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 10 monclient(hunting): my global_id is 4139 2024-03-29T11:19:25.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 1 -- 172.21.15.119:0/164331844 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72480b8960 con 0x7f724809ac40 2024-03-29T11:19:25.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 1 -- 172.21.15.119:0/164331844 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1077481154 0 0) 0x7f72480b7520 con 0x7f72481461c0 2024-03-29T11:19:25.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:25.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:25.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:25.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 10 monclient(hunting): my global_id is 4143 2024-03-29T11:19:25.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 1 -- 172.21.15.119:0/164331844 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72480b7d30 con 0x7f72481461c0 2024-03-29T11:19:25.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 1 -- 172.21.15.119:0/164331844 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4091567418 0 0) 0x7f722c004eb0 con 0x7f72480b63e0 2024-03-29T11:19:25.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 1 -- 172.21.15.119:0/164331844 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72480b7520 con 0x7f72480b63e0 2024-03-29T11:19:25.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 1 -- 172.21.15.119:0/164331844 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f7238001a10 con 0x7f72480b63e0 2024-03-29T11:19:25.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:25.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:25.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 10 monclient(hunting): dump: 2024-03-29T11:19:25.920 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 1 2024-03-29T11:19:25.920 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:25.920 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.920 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.920 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:25.920 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:25.920 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:25.920 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:25.920 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:25.921 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:25.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:25.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 1 -- 172.21.15.119:0/164331844 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2491596223 0 0) 0x7f72480b8960 con 0x7f724809ac40 2024-03-29T11:19:25.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 1 -- 172.21.15.119:0/164331844 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f722c004eb0 con 0x7f724809ac40 2024-03-29T11:19:25.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 1 -- 172.21.15.119:0/164331844 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 621912759 0 0) 0x7f72480b7d30 con 0x7f72481461c0 2024-03-29T11:19:25.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 1 -- 172.21.15.119:0/164331844 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72480b8960 con 0x7f72481461c0 2024-03-29T11:19:25.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 1 -- 172.21.15.119:0/164331844 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f7244001920 con 0x7f724809ac40 2024-03-29T11:19:25.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:25.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:25.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 10 monclient(hunting): dump: 2024-03-29T11:19:25.921 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 1 2024-03-29T11:19:25.921 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:25.921 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.921 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.921 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:25.921 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:25.922 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:25.922 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:25.922 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:25.922 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:25.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 1 -- 172.21.15.119:0/164331844 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f7240001f00 con 0x7f72481461c0 2024-03-29T11:19:25.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:25.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:25.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 10 monclient(hunting): dump: 2024-03-29T11:19:25.922 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 1 2024-03-29T11:19:25.922 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:25.922 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.922 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.922 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:25.922 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:25.922 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:25.923 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:25.923 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:25.923 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:25.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 1 -- 172.21.15.119:0/164331844 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1518506845 0 0) 0x7f72480b7520 con 0x7f72480b63e0 2024-03-29T11:19:25.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 1 -- 172.21.15.119:0/164331844 >> v1:172.21.15.104:6790/0 conn(0x7f724809ac40 legacy=0x7f72480b8250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:25.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 1 -- 172.21.15.119:0/164331844 >> v1:172.21.15.104:6789/0 conn(0x7f72481461c0 legacy=0x7f72481e2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:25.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:19:25.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 1 monclient: found mon.b 2024-03-29T11:19:25.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:25.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:19:25.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 1 -- 172.21.15.119:0/164331844 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72481d7470 con 0x7f72480b63e0 2024-03-29T11:19:25.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 10 monclient: _finish_auth 0 2024-03-29T11:19:25.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:19:25.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:25.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f724f425640 5 monclient: authenticate success, global_id 4129 2024-03-29T11:19:25.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f724f425640 10 monclient: _renew_subs 2024-03-29T11:19:25.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f724f425640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:19:25.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f724f425640 1 -- 172.21.15.119:0/164331844 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72481d76e0 con 0x7f72480b63e0 2024-03-29T11:19:25.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 1 -- 172.21.15.119:0/164331844 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7238001cf0 con 0x7f72480b63e0 2024-03-29T11:19:25.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:25.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 1 -- 172.21.15.119:0/164331844 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f7238003a60 con 0x7f72480b63e0 2024-03-29T11:19:25.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:25.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:25.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 10 monclient: dump: 2024-03-29T11:19:25.924 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 1 2024-03-29T11:19:25.924 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:25.924 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.924 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:25.924 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:25.924 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:25.924 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:25.925 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:25.925 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:25.925 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:25.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723dffb640 1 -- 172.21.15.119:0/164331844 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f7238003fd0 con 0x7f72480b63e0 2024-03-29T11:19:25.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.928+0000 7f723f7fe640 4 set_mon_vals no callback set 2024-03-29T11:19:25.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.929+0000 7f724f425640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:25.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.929+0000 7f724f425640 10 monclient: _renew_subs 2024-03-29T11:19:25.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.929+0000 7f724f425640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:19:25.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.929+0000 7f724f425640 1 -- 172.21.15.119:0/164331844 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f72481d4e20 con 0x7f72480b63e0 2024-03-29T11:19:25.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.929+0000 7f724f425640 1 librados: init done 2024-03-29T11:19:25.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.929+0000 7f723dffb640 1 -- 172.21.15.119:0/164331844 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 1394902812 0 0) 0x7f7238005cd0 con 0x7f72480b63e0 2024-03-29T11:19:25.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.929+0000 7f723dffb640 10 client.4129.objecter ms_dispatch 0x7f7248000b90 osd_map(6..6 src has 1..6) v4 2024-03-29T11:19:25.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.929+0000 7f723dffb640 3 client.4129.objecter handle_osd_map got epochs [6,6] > 0 2024-03-29T11:19:25.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.929+0000 7f723dffb640 3 client.4129.objecter handle_osd_map decoding full epoch 6 2024-03-29T11:19:25.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.929+0000 7f723dffb640 20 client.4129.objecter dump_active .. 0 homeless 2024-03-29T11:19:25.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.929+0000 7f724f425640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:25.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.929+0000 7f724f425640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:25.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.929+0000 7f724f425640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:19:25.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.929+0000 7f724f425640 1 -- 172.21.15.119:0/164331844 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72480b7b90 con 0x7f72480b63e0 2024-03-29T11:19:25.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.933+0000 7f723dffb640 1 -- 172.21.15.119:0/164331844 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f72480b7b90 con 0x7f72480b63e0 2024-03-29T11:19:25.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.933+0000 7f723dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:25.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:25.933+0000 7f723dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:25.930 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-03-29T11:19:25.953+0000 7f32f3443040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:19:25.947 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-29T11:19:25.971+0000 7f0ede043040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:19:25.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.006+0000 7f724f425640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "010f8945-19c0-4f55-b2fc-21d83b076f4e", "id": 4}] 2024-03-29T11:19:25.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.006+0000 7f724f425640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "010f8945-19c0-4f55-b2fc-21d83b076f4e", "id": 4}] 2024-03-29T11:19:25.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.006+0000 7f724f425640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:19:25.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.006+0000 7f724f425640 1 -- 172.21.15.119:0/164331844 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "osd new", "uuid": "010f8945-19c0-4f55-b2fc-21d83b076f4e", "id": 4} v 0) v1 -- 0x7f72481d5070 con 0x7f72480b63e0 2024-03-29T11:19:25.984 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-03-29T11:19:26.008+0000 7f32f3443040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:19:25.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.010+0000 7f723dffb640 1 -- 172.21.15.119:0/164331844 <== mon.1 v1:172.21.15.119:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "010f8945-19c0-4f55-b2fc-21d83b076f4e", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 3665878808 0 1697012583) 0x7f72380057c0 con 0x7f72480b63e0 2024-03-29T11:19:25.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.010+0000 7f723dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "010f8945-19c0-4f55-b2fc-21d83b076f4e", "id": 4}] 2024-03-29T11:19:25.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.010+0000 7f723dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:25.987 INFO:teuthology.orchestra.run.smithi119.stdout:4 2024-03-29T11:19:25.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.011+0000 7f724f425640 10 librados: watch_flush enter 2024-03-29T11:19:25.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.011+0000 7f724f425640 10 librados: watch_flush exit 2024-03-29T11:19:25.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.011+0000 7f724f425640 20 client.4129.objecter shutdown clearing up homeless session... 2024-03-29T11:19:25.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.011+0000 7f724f425640 10 client.4129.objecter successfully canceled tick 2024-03-29T11:19:25.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.011+0000 7f724f425640 10 monclient: shutdown 2024-03-29T11:19:25.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.011+0000 7f724f425640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:25.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.011+0000 7f724f425640 1 -- 172.21.15.119:0/164331844 >> v1:172.21.15.119:6789/0 conn(0x7f72480b63e0 legacy=0x7f72481d4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:25.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.011+0000 7f724f425640 1 -- 172.21.15.119:0/164331844 shutdown_connections 2024-03-29T11:19:25.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.011+0000 7f724f425640 1 -- 172.21.15.119:0/164331844 >> 172.21.15.119:0/164331844 conn(0x7f7248066b00 msgr2=0x7f724813d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:25.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.011+0000 7f724f425640 1 -- 172.21.15.119:0/164331844 shutdown_connections 2024-03-29T11:19:25.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.011+0000 7f724f425640 1 -- 172.21.15.119:0/164331844 wait complete. 2024-03-29T11:19:25.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.011+0000 7f724f425640 1 librados: shutdown 2024-03-29T11:19:25.995 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-29T11:19:26.019+0000 7f0ede043040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:19:25.999 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster ceph osd new 123ab874-0241-4d9a-b9ff-e35e7bb09898 5 2024-03-29T11:19:26.035 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-03-29T11:19:26.058+0000 7f32f3443040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:19:26.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.157+0000 7ff3f21b5640 10 monclient: get_monmap_and_config 2024-03-29T11:19:26.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.157+0000 7ff3f21b5640 10 monclient: build_initial_monmap 2024-03-29T11:19:26.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.158+0000 7ff3f21b5640 10 monclient: monmap: 2024-03-29T11:19:26.135 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 0 2024-03-29T11:19:26.135 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:26.135 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:19:26.159201+0000 2024-03-29T11:19:26.135 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:19:26.159201+0000 2024-03-29T11:19:26.135 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:26.135 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:26.135 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:19:26.136 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:19:26.136 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:19:26.136 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:26.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.158+0000 7ff3f21b5640 1 Processor -- start 2024-03-29T11:19:26.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.158+0000 7ff3f21b5640 1 -- start start 2024-03-29T11:19:26.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.158+0000 7ff3f21b5640 10 monclient: init 2024-03-29T11:19:26.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.159+0000 7ff3f21b5640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:26.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.159+0000 7ff3f21b5640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:26.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.159+0000 7ff3f21b5640 10 monclient(hunting): picked mon.noname-b con 0x7ff3ec09ac40 addr v1:172.21.15.104:6790/0 2024-03-29T11:19:26.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.159+0000 7ff3f21b5640 10 monclient(hunting): picked mon.noname-c con 0x7ff3ec09c870 addr v1:172.21.15.119:6789/0 2024-03-29T11:19:26.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.159+0000 7ff3f21b5640 10 monclient(hunting): picked mon.noname-a con 0x7ff3ec0b4360 addr v1:172.21.15.104:6789/0 2024-03-29T11:19:26.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.159+0000 7ff3f21b5640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3ec09dec0 con 0x7ff3ec0b4360 2024-03-29T11:19:26.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.159+0000 7ff3f21b5640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3ec09da30 con 0x7ff3ec09c870 2024-03-29T11:19:26.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.159+0000 7ff3f21b5640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3ec0b39c0 con 0x7ff3ec09ac40 2024-03-29T11:19:26.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.159+0000 7ff3f21b5640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:26.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.159+0000 7ff3f21b5640 10 monclient(hunting): authenticate will time out at 943.436441s 2024-03-29T11:19:26.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.159+0000 7ff3eaffd640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7ff3ec09c870 0x7ff3ec13f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:48412/0 (socket says 172.21.15.119:48412) 2024-03-29T11:19:26.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.159+0000 7ff3eaffd640 1 -- 172.21.15.119:0/1467056702 learned_addr learned my addr 172.21.15.119:0/1467056702 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-03-29T11:19:26.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.159+0000 7ff3ea7fc640 1 -- 172.21.15.119:0/1467056702 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2553392101 0 0) 0x7ff3ec09da30 con 0x7ff3ec09c870 2024-03-29T11:19:26.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.159+0000 7ff3ea7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:26.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.159+0000 7ff3ea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:26.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.159+0000 7ff3ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:26.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.159+0000 7ff3ea7fc640 10 monclient(hunting): my global_id is 4132 2024-03-29T11:19:26.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.159+0000 7ff3ea7fc640 1 -- 172.21.15.119:0/1467056702 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3d4003520 con 0x7ff3ec09c870 2024-03-29T11:19:26.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.159+0000 7ff3ea7fc640 1 -- 172.21.15.119:0/1467056702 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1432938826 0 0) 0x7ff3e4001490 con 0x7ff3ec0b4360 2024-03-29T11:19:26.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.159+0000 7ff3ea7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:26.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.159+0000 7ff3ea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:26.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.159+0000 7ff3ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:26.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.159+0000 7ff3ea7fc640 10 monclient(hunting): my global_id is 4146 2024-03-29T11:19:26.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.159+0000 7ff3ea7fc640 1 -- 172.21.15.119:0/1467056702 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3ec09da30 con 0x7ff3ec0b4360 2024-03-29T11:19:26.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.159+0000 7ff3ea7fc640 1 -- 172.21.15.119:0/1467056702 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3283802895 0 0) 0x7ff3ec0b39c0 con 0x7ff3ec09ac40 2024-03-29T11:19:26.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.159+0000 7ff3ea7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:26.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.159+0000 7ff3ea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:26.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.159+0000 7ff3ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:26.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.159+0000 7ff3ea7fc640 10 monclient(hunting): my global_id is 4142 2024-03-29T11:19:26.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.159+0000 7ff3ea7fc640 1 -- 172.21.15.119:0/1467056702 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3e4001490 con 0x7ff3ec09ac40 2024-03-29T11:19:26.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.160+0000 7ff3ea7fc640 1 -- 172.21.15.119:0/1467056702 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 963162685 0 0) 0x7ff3d4003520 con 0x7ff3ec09c870 2024-03-29T11:19:26.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.160+0000 7ff3ea7fc640 1 -- 172.21.15.119:0/1467056702 >> v1:172.21.15.104:6790/0 conn(0x7ff3ec09ac40 legacy=0x7ff3ec0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:26.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.160+0000 7ff3ea7fc640 1 -- 172.21.15.119:0/1467056702 >> v1:172.21.15.104:6789/0 conn(0x7ff3ec0b4360 legacy=0x7ff3ec14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:26.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.160+0000 7ff3ea7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:19:26.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.160+0000 7ff3ea7fc640 1 monclient: found mon.noname-c 2024-03-29T11:19:26.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.160+0000 7ff3ea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:26.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.160+0000 7ff3ea7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.119:6789/0 2024-03-29T11:19:26.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.160+0000 7ff3ea7fc640 1 -- 172.21.15.119:0/1467056702 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3ec0b5160 con 0x7ff3ec09c870 2024-03-29T11:19:26.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.160+0000 7ff3ea7fc640 10 monclient: _finish_auth 0 2024-03-29T11:19:26.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.160+0000 7ff3ea7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:19:26.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.160+0000 7ff3ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:26.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.160+0000 7ff3ea7fc640 1 -- 172.21.15.119:0/1467056702 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7ff3dc001690 con 0x7ff3ec09c870 2024-03-29T11:19:26.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.160+0000 7ff3f21b5640 5 monclient: authenticate success, global_id 4132 2024-03-29T11:19:26.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.160+0000 7ff3f21b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:26.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.160+0000 7ff3ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:26.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.160+0000 7ff3ea7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:26.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.160+0000 7ff3ea7fc640 10 monclient: dump: 2024-03-29T11:19:26.148 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 1 2024-03-29T11:19:26.148 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:26.148 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:26.148 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:26.148 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:26.148 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:26.148 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:26.149 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:26.149 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:26.149 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:26.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.160+0000 7ff3f21b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:26.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.160+0000 7ff3ea7fc640 1 -- 172.21.15.119:0/1467056702 <== mon.1 v1:172.21.15.119:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3dc001870 con 0x7ff3ec09c870 2024-03-29T11:19:26.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.160+0000 7ff3ea7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:26.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.160+0000 7ff3f21b5640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:26.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.160+0000 7ff3ea7fc640 1 -- 172.21.15.119:0/1467056702 <== mon.1 v1:172.21.15.119:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7ff3dc001c80 con 0x7ff3ec09c870 2024-03-29T11:19:26.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.160+0000 7ff3f21b5640 4 set_mon_vals no callback set 2024-03-29T11:19:26.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.160+0000 7ff3ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:26.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.160+0000 7ff3ea7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:26.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.160+0000 7ff3ea7fc640 10 monclient: dump: 2024-03-29T11:19:26.149 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 1 2024-03-29T11:19:26.149 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:26.150 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:26.150 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:26.150 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:26.150 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:26.150 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:26.150 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:26.150 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:26.150 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:26.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.160+0000 7ff3f21b5640 10 monclient: shutdown 2024-03-29T11:19:26.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.160+0000 7ff3f21b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:26.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.160+0000 7ff3f21b5640 1 -- 172.21.15.119:0/1467056702 >> v1:172.21.15.119:6789/0 conn(0x7ff3ec09c870 legacy=0x7ff3ec13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:26.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.160+0000 7ff3f21b5640 1 -- 172.21.15.119:0/1467056702 shutdown_connections 2024-03-29T11:19:26.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.160+0000 7ff3f21b5640 1 -- 172.21.15.119:0/1467056702 >> 172.21.15.119:0/1467056702 conn(0x7ff3ec066b00 msgr2=0x7ff3ec066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:26.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.160+0000 7ff3f21b5640 1 -- 172.21.15.119:0/1467056702 shutdown_connections 2024-03-29T11:19:26.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.160+0000 7ff3f21b5640 1 -- 172.21.15.119:0/1467056702 wait complete. 2024-03-29T11:19:26.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.161+0000 7ff3f21b5640 10 monclient: build_initial_monmap 2024-03-29T11:19:26.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.161+0000 7ff3f21b5640 10 monclient: monmap: 2024-03-29T11:19:26.151 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 0 2024-03-29T11:19:26.151 INFO:teuthology.orchestra.run.smithi119.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:26.151 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 0.000000 2024-03-29T11:19:26.151 INFO:teuthology.orchestra.run.smithi119.stderr:created 0.000000 2024-03-29T11:19:26.151 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:26.151 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:26.151 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:19:26.151 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:19:26.151 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:19:26.151 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:26.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.162+0000 7ff3f21b5640 1 librados: starting msgr at 2024-03-29T11:19:26.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.162+0000 7ff3f21b5640 1 librados: starting objecter 2024-03-29T11:19:26.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.162+0000 7ff3f21b5640 1 Processor -- start 2024-03-29T11:19:26.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.162+0000 7ff3f21b5640 1 -- start start 2024-03-29T11:19:26.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.162+0000 7ff3f21b5640 1 librados: setting wanted keys 2024-03-29T11:19:26.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.162+0000 7ff3f21b5640 1 librados: calling monclient init 2024-03-29T11:19:26.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.162+0000 7ff3f21b5640 10 monclient: init 2024-03-29T11:19:26.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.163+0000 7ff3f21b5640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:26.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.163+0000 7ff3f21b5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:26.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.163+0000 7ff3f21b5640 10 monclient(hunting): picked mon.noname-a con 0x7ff3ec09ac40 addr v1:172.21.15.104:6789/0 2024-03-29T11:19:26.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.163+0000 7ff3f21b5640 10 monclient(hunting): picked mon.noname-c con 0x7ff3ec0af4e0 addr v1:172.21.15.104:6790/0 2024-03-29T11:19:26.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.163+0000 7ff3f21b5640 10 monclient(hunting): picked mon.noname-b con 0x7ff3ec1d7cf0 addr v1:172.21.15.119:6789/0 2024-03-29T11:19:26.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.163+0000 7ff3f21b5640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3ec0b1940 con 0x7ff3ec09ac40 2024-03-29T11:19:26.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.163+0000 7ff3f21b5640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3ec1d70c0 con 0x7ff3ec1d7cf0 2024-03-29T11:19:26.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.163+0000 7ff3f21b5640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3ec0acd00 con 0x7ff3ec0af4e0 2024-03-29T11:19:26.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.163+0000 7ff3f21b5640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:26.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.163+0000 7ff3f21b5640 10 monclient(hunting): authenticate will time out at 943.440397s 2024-03-29T11:19:26.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.163+0000 7ff3ebfff640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7ff3ec1d7cf0 0x7ff3ec1e1b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:48422/0 (socket says 172.21.15.119:48422) 2024-03-29T11:19:26.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.163+0000 7ff3ebfff640 1 -- 172.21.15.119:0/1702539360 learned_addr learned my addr 172.21.15.119:0/1702539360 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-03-29T11:19:26.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.163+0000 7ff3d3fff640 10 client.?.objecter ms_handle_connect 0x7ff3ec1d7cf0 2024-03-29T11:19:26.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.163+0000 7ff3d3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:26.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.163+0000 7ff3d3fff640 10 client.?.objecter ms_handle_connect 0x7ff3ec09ac40 2024-03-29T11:19:26.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.163+0000 7ff3d3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:26.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.163+0000 7ff3d3fff640 10 client.?.objecter ms_handle_connect 0x7ff3ec0af4e0 2024-03-29T11:19:26.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.163+0000 7ff3d3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:26.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.163+0000 7ff3d3fff640 1 -- 172.21.15.119:0/1702539360 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2720908168 0 0) 0x7ff3ec1d70c0 con 0x7ff3ec1d7cf0 2024-03-29T11:19:26.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.163+0000 7ff3d3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:26.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.163+0000 7ff3d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:26.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.163+0000 7ff3d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:26.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.163+0000 7ff3d3fff640 10 monclient(hunting): my global_id is 4135 2024-03-29T11:19:26.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.163+0000 7ff3d3fff640 1 -- 172.21.15.119:0/1702539360 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3d8004eb0 con 0x7ff3ec1d7cf0 2024-03-29T11:19:26.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.163+0000 7ff3d3fff640 1 -- 172.21.15.119:0/1702539360 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 714709017 0 0) 0x7ff3ec0b1940 con 0x7ff3ec09ac40 2024-03-29T11:19:26.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.163+0000 7ff3d3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:26.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.163+0000 7ff3d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:26.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.163+0000 7ff3d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:26.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.163+0000 7ff3d3fff640 10 monclient(hunting): my global_id is 4149 2024-03-29T11:19:26.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.163+0000 7ff3d3fff640 1 -- 172.21.15.119:0/1702539360 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3ec1d70c0 con 0x7ff3ec09ac40 2024-03-29T11:19:26.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.163+0000 7ff3d3fff640 1 -- 172.21.15.119:0/1702539360 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 239529237 0 0) 0x7ff3ec0acd00 con 0x7ff3ec0af4e0 2024-03-29T11:19:26.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.163+0000 7ff3d3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:26.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.163+0000 7ff3d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:26.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.163+0000 7ff3d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:26.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.163+0000 7ff3d3fff640 10 monclient(hunting): my global_id is 4145 2024-03-29T11:19:26.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.163+0000 7ff3d3fff640 1 -- 172.21.15.119:0/1702539360 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3ec0b1940 con 0x7ff3ec0af4e0 2024-03-29T11:19:26.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.163+0000 7ff3d3fff640 1 -- 172.21.15.119:0/1702539360 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 96044507 0 0) 0x7ff3d8004eb0 con 0x7ff3ec1d7cf0 2024-03-29T11:19:26.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.163+0000 7ff3d3fff640 1 -- 172.21.15.119:0/1702539360 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3ec0acd00 con 0x7ff3ec1d7cf0 2024-03-29T11:19:26.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.163+0000 7ff3d3fff640 1 -- 172.21.15.119:0/1702539360 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7ff3e4001c80 con 0x7ff3ec1d7cf0 2024-03-29T11:19:26.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.163+0000 7ff3d3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:26.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.163+0000 7ff3d3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:26.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.163+0000 7ff3d3fff640 10 monclient(hunting): dump: 2024-03-29T11:19:26.155 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 1 2024-03-29T11:19:26.155 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:26.155 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:26.155 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:26.155 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:26.155 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:26.155 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:26.155 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:26.155 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:26.156 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:26.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.163+0000 7ff3d3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:26.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.163+0000 7ff3d3fff640 1 -- 172.21.15.119:0/1702539360 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3513430031 0 0) 0x7ff3ec1d70c0 con 0x7ff3ec09ac40 2024-03-29T11:19:26.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.164+0000 7ff3d3fff640 1 -- 172.21.15.119:0/1702539360 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3d8004eb0 con 0x7ff3ec09ac40 2024-03-29T11:19:26.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.164+0000 7ff3d3fff640 1 -- 172.21.15.119:0/1702539360 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7ff3e00018b0 con 0x7ff3ec09ac40 2024-03-29T11:19:26.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.164+0000 7ff3d3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:26.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.164+0000 7ff3d3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:26.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.164+0000 7ff3d3fff640 10 monclient(hunting): dump: 2024-03-29T11:19:26.157 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 1 2024-03-29T11:19:26.157 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:26.157 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:26.157 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:26.158 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:26.158 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:26.158 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:26.158 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:26.158 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:26.158 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:26.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.164+0000 7ff3d3fff640 1 -- 172.21.15.119:0/1702539360 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2773187256 0 0) 0x7ff3ec0b1940 con 0x7ff3ec0af4e0 2024-03-29T11:19:26.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.164+0000 7ff3d3fff640 1 -- 172.21.15.119:0/1702539360 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3ec1d70c0 con 0x7ff3ec0af4e0 2024-03-29T11:19:26.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.164+0000 7ff3d3fff640 1 -- 172.21.15.119:0/1702539360 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7ff3dc001b80 con 0x7ff3ec0af4e0 2024-03-29T11:19:26.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.164+0000 7ff3d3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:26.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.164+0000 7ff3d3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:26.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.164+0000 7ff3d3fff640 10 monclient(hunting): dump: 2024-03-29T11:19:26.159 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 1 2024-03-29T11:19:26.159 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:26.159 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:26.159 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:26.159 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:26.159 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:26.159 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:26.159 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:26.159 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:26.159 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:26.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.164+0000 7ff3d3fff640 1 -- 172.21.15.119:0/1702539360 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 726394474 0 0) 0x7ff3ec0acd00 con 0x7ff3ec1d7cf0 2024-03-29T11:19:26.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.164+0000 7ff3d3fff640 1 -- 172.21.15.119:0/1702539360 >> v1:172.21.15.104:6790/0 conn(0x7ff3ec0af4e0 legacy=0x7ff3ec1d75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:26.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.164+0000 7ff3d3fff640 1 -- 172.21.15.119:0/1702539360 >> v1:172.21.15.104:6789/0 conn(0x7ff3ec09ac40 legacy=0x7ff3ec0b03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:26.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.164+0000 7ff3d3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:19:26.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.164+0000 7ff3d3fff640 1 monclient: found mon.b 2024-03-29T11:19:26.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.164+0000 7ff3d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:26.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.164+0000 7ff3d3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:19:26.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.164+0000 7ff3d3fff640 1 -- 172.21.15.119:0/1702539360 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3ec0af970 con 0x7ff3ec1d7cf0 2024-03-29T11:19:26.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.164+0000 7ff3d3fff640 10 monclient: _finish_auth 0 2024-03-29T11:19:26.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.164+0000 7ff3d3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:19:26.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.164+0000 7ff3d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:26.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.164+0000 7ff3f21b5640 5 monclient: authenticate success, global_id 4135 2024-03-29T11:19:26.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.164+0000 7ff3f21b5640 10 monclient: _renew_subs 2024-03-29T11:19:26.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.164+0000 7ff3f21b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:19:26.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.164+0000 7ff3f21b5640 1 -- 172.21.15.119:0/1702539360 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3ec1e3260 con 0x7ff3ec1d7cf0 2024-03-29T11:19:26.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.164+0000 7ff3f21b5640 10 client.4135.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:26.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.164+0000 7ff3f21b5640 10 monclient: _renew_subs 2024-03-29T11:19:26.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.164+0000 7ff3f21b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:19:26.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.164+0000 7ff3f21b5640 1 -- 172.21.15.119:0/1702539360 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3ec1e35d0 con 0x7ff3ec1d7cf0 2024-03-29T11:19:26.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.164+0000 7ff3f21b5640 1 librados: init done 2024-03-29T11:19:26.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.164+0000 7ff3d3fff640 1 -- 172.21.15.119:0/1702539360 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3e4003970 con 0x7ff3ec1d7cf0 2024-03-29T11:19:26.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.164+0000 7ff3d3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:26.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.164+0000 7ff3d3fff640 1 -- 172.21.15.119:0/1702539360 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7ff3e4003cc0 con 0x7ff3ec1d7cf0 2024-03-29T11:19:26.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.164+0000 7ff3d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:26.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.164+0000 7ff3d3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:26.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.164+0000 7ff3e9ffb640 4 set_mon_vals no callback set 2024-03-29T11:19:26.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.164+0000 7ff3d3fff640 10 monclient: dump: 2024-03-29T11:19:26.161 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 1 2024-03-29T11:19:26.161 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:26.161 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:26.161 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:26.162 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:26.162 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:26.162 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:26.162 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:26.162 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:26.162 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:26.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.164+0000 7ff3d3fff640 1 -- 172.21.15.119:0/1702539360 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7ff3e40041b0 con 0x7ff3ec1d7cf0 2024-03-29T11:19:26.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.164+0000 7ff3d3fff640 1 -- 172.21.15.119:0/1702539360 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 1968528 0 0) 0x7ff3e4004460 con 0x7ff3ec1d7cf0 2024-03-29T11:19:26.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.164+0000 7ff3d3fff640 10 client.4135.objecter ms_dispatch 0x7ff3ec000b90 osd_map(7..7 src has 1..7) v4 2024-03-29T11:19:26.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.164+0000 7ff3d3fff640 3 client.4135.objecter handle_osd_map got epochs [7,7] > 0 2024-03-29T11:19:26.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.164+0000 7ff3d3fff640 3 client.4135.objecter handle_osd_map decoding full epoch 7 2024-03-29T11:19:26.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.164+0000 7ff3d3fff640 20 client.4135.objecter dump_active .. 0 homeless 2024-03-29T11:19:26.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.165+0000 7ff3f21b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:26.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.165+0000 7ff3f21b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:26.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.165+0000 7ff3f21b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:19:26.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.165+0000 7ff3f21b5640 1 -- 172.21.15.119:0/1702539360 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3cc005080 con 0x7ff3ec1d7cf0 2024-03-29T11:19:26.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.167+0000 7ff3d3fff640 1 -- 172.21.15.119:0/1702539360 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ff3e40046f0 con 0x7ff3ec1d7cf0 2024-03-29T11:19:26.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.167+0000 7ff3d3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:26.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.167+0000 7ff3d3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:26.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.239+0000 7ff3f21b5640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "123ab874-0241-4d9a-b9ff-e35e7bb09898", "id": 5}] 2024-03-29T11:19:26.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.239+0000 7ff3f21b5640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "123ab874-0241-4d9a-b9ff-e35e7bb09898", "id": 5}] 2024-03-29T11:19:26.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.239+0000 7ff3f21b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:19:26.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.239+0000 7ff3f21b5640 1 -- 172.21.15.119:0/1702539360 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "osd new", "uuid": "123ab874-0241-4d9a-b9ff-e35e7bb09898", "id": 5} v 0) v1 -- 0x7ff3cc0058b0 con 0x7ff3ec1d7cf0 2024-03-29T11:19:26.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.243+0000 7ff3d3fff640 1 -- 172.21.15.119:0/1702539360 <== mon.1 v1:172.21.15.119:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "123ab874-0241-4d9a-b9ff-e35e7bb09898", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 1999968625 0 1988412176) 0x7ff3e4005c10 con 0x7ff3ec1d7cf0 2024-03-29T11:19:26.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.243+0000 7ff3d3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "123ab874-0241-4d9a-b9ff-e35e7bb09898", "id": 5}] 2024-03-29T11:19:26.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.243+0000 7ff3d3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:26.221 INFO:teuthology.orchestra.run.smithi119.stdout:5 2024-03-29T11:19:26.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.244+0000 7ff3f21b5640 10 librados: watch_flush enter 2024-03-29T11:19:26.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.244+0000 7ff3f21b5640 10 librados: watch_flush exit 2024-03-29T11:19:26.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.244+0000 7ff3f21b5640 20 client.4135.objecter shutdown clearing up homeless session... 2024-03-29T11:19:26.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.244+0000 7ff3f21b5640 10 client.4135.objecter successfully canceled tick 2024-03-29T11:19:26.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.244+0000 7ff3f21b5640 10 monclient: shutdown 2024-03-29T11:19:26.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.244+0000 7ff3f21b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:26.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.244+0000 7ff3f21b5640 1 -- 172.21.15.119:0/1702539360 >> v1:172.21.15.119:6789/0 conn(0x7ff3ec1d7cf0 legacy=0x7ff3ec1e1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:26.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.244+0000 7ff3f21b5640 1 -- 172.21.15.119:0/1702539360 shutdown_connections 2024-03-29T11:19:26.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.244+0000 7ff3f21b5640 1 -- 172.21.15.119:0/1702539360 >> 172.21.15.119:0/1702539360 conn(0x7ff3ec066b00 msgr2=0x7ff3ec13d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:26.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.244+0000 7ff3f21b5640 1 -- 172.21.15.119:0/1702539360 shutdown_connections 2024-03-29T11:19:26.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.244+0000 7ff3f21b5640 1 -- 172.21.15.119:0/1702539360 wait complete. 2024-03-29T11:19:26.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.245+0000 7ff3f21b5640 1 librados: shutdown 2024-03-29T11:19:26.231 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster ceph osd new 9c0a3b9b-48bb-4cd2-a973-ab112c32f77f 6 2024-03-29T11:19:26.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.354+0000 7f7a3e72b640 10 monclient: get_monmap_and_config 2024-03-29T11:19:26.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.354+0000 7f7a3e72b640 10 monclient: build_initial_monmap 2024-03-29T11:19:26.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.354+0000 7f7a3e72b640 10 monclient: monmap: 2024-03-29T11:19:26.332 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 0 2024-03-29T11:19:26.332 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:26.332 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:19:26.355883+0000 2024-03-29T11:19:26.332 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:19:26.355883+0000 2024-03-29T11:19:26.332 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:26.332 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:26.332 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:19:26.332 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:19:26.332 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:19:26.332 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:26.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.355+0000 7f7a3e72b640 1 Processor -- start 2024-03-29T11:19:26.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.355+0000 7f7a3e72b640 1 -- start start 2024-03-29T11:19:26.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.355+0000 7f7a3e72b640 10 monclient: init 2024-03-29T11:19:26.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.355+0000 7f7a3e72b640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:26.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.355+0000 7f7a3e72b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:19:26.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.355+0000 7f7a3e72b640 10 monclient(hunting): picked mon.noname-b con 0x7f7a3809ac40 addr v1:172.21.15.104:6790/0 2024-03-29T11:19:26.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.355+0000 7f7a3e72b640 10 monclient(hunting): picked mon.noname-a con 0x7f7a3809c870 addr v1:172.21.15.104:6789/0 2024-03-29T11:19:26.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a3e72b640 10 monclient(hunting): picked mon.noname-c con 0x7f7a380b4360 addr v1:172.21.15.119:6789/0 2024-03-29T11:19:26.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a3e72b640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a3809dec0 con 0x7f7a3809c870 2024-03-29T11:19:26.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a3e72b640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a3809da30 con 0x7f7a380b4360 2024-03-29T11:19:26.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a3e72b640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a380b39c0 con 0x7f7a3809ac40 2024-03-29T11:19:26.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a3e72b640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:26.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a3e72b640 10 monclient(hunting): authenticate will time out at 943.633087s 2024-03-29T11:19:26.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a3ccc8640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f7a380b4360 0x7f7a3814d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:48428/0 (socket says 172.21.15.119:48428) 2024-03-29T11:19:26.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a3ccc8640 1 -- 172.21.15.119:0/3053614359 learned_addr learned my addr 172.21.15.119:0/3053614359 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-03-29T11:19:26.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a36ffd640 1 -- 172.21.15.119:0/3053614359 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4239462591 0 0) 0x7f7a3809da30 con 0x7f7a380b4360 2024-03-29T11:19:26.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a36ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:26.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a36ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:26.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:26.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a36ffd640 10 monclient(hunting): my global_id is 4138 2024-03-29T11:19:26.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a36ffd640 1 -- 172.21.15.119:0/3053614359 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a20003520 con 0x7f7a380b4360 2024-03-29T11:19:26.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a36ffd640 1 -- 172.21.15.119:0/3053614359 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1041233823 0 0) 0x7f7a28001110 con 0x7f7a3809c870 2024-03-29T11:19:26.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a36ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:26.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a36ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:26.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:26.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a36ffd640 10 monclient(hunting): my global_id is 4152 2024-03-29T11:19:26.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a36ffd640 1 -- 172.21.15.119:0/3053614359 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a3809da30 con 0x7f7a3809c870 2024-03-29T11:19:26.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a36ffd640 1 -- 172.21.15.119:0/3053614359 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3073492882 0 0) 0x7f7a380b39c0 con 0x7f7a3809ac40 2024-03-29T11:19:26.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a36ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:26.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a36ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:26.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:26.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a36ffd640 10 monclient(hunting): my global_id is 4148 2024-03-29T11:19:26.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a36ffd640 1 -- 172.21.15.119:0/3053614359 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a28001110 con 0x7f7a3809ac40 2024-03-29T11:19:26.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a36ffd640 1 -- 172.21.15.119:0/3053614359 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1141932035 0 0) 0x7f7a20003520 con 0x7f7a380b4360 2024-03-29T11:19:26.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a36ffd640 1 -- 172.21.15.119:0/3053614359 >> v1:172.21.15.104:6790/0 conn(0x7f7a3809ac40 legacy=0x7f7a380b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:26.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a36ffd640 1 -- 172.21.15.119:0/3053614359 >> v1:172.21.15.104:6789/0 conn(0x7f7a3809c870 legacy=0x7f7a3813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:26.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a36ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:26.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a36ffd640 1 monclient: found mon.noname-c 2024-03-29T11:19:26.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a36ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:26.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a36ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.119:6789/0 2024-03-29T11:19:26.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a36ffd640 1 -- 172.21.15.119:0/3053614359 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a380b5160 con 0x7f7a380b4360 2024-03-29T11:19:26.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a36ffd640 10 monclient: _finish_auth 0 2024-03-29T11:19:26.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a36ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:26.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:26.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a3e72b640 5 monclient: authenticate success, global_id 4138 2024-03-29T11:19:26.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a36ffd640 1 -- 172.21.15.119:0/3053614359 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f7a30001a10 con 0x7f7a380b4360 2024-03-29T11:19:26.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a3e72b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:26.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:26.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a36ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:26.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a36ffd640 10 monclient: dump: 2024-03-29T11:19:26.346 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 1 2024-03-29T11:19:26.346 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:26.347 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:26.347 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:26.347 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:26.347 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:26.347 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:26.347 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:26.347 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:26.347 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:26.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a3e72b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:26.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a36ffd640 1 -- 172.21.15.119:0/3053614359 <== mon.1 v1:172.21.15.119:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a30001bf0 con 0x7f7a380b4360 2024-03-29T11:19:26.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a36ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:26.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a3e72b640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:26.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a36ffd640 1 -- 172.21.15.119:0/3053614359 <== mon.1 v1:172.21.15.119:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f7a30002000 con 0x7f7a380b4360 2024-03-29T11:19:26.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:26.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a3e72b640 4 set_mon_vals no callback set 2024-03-29T11:19:26.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a36ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:26.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.356+0000 7f7a36ffd640 10 monclient: dump: 2024-03-29T11:19:26.348 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 1 2024-03-29T11:19:26.348 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:26.348 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:26.348 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:26.348 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:26.348 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:26.348 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:26.348 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:26.348 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:26.348 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:26.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.357+0000 7f7a3e72b640 10 monclient: shutdown 2024-03-29T11:19:26.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.357+0000 7f7a3e72b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:26.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.357+0000 7f7a3e72b640 1 -- 172.21.15.119:0/3053614359 >> v1:172.21.15.119:6789/0 conn(0x7f7a380b4360 legacy=0x7f7a3814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:26.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.357+0000 7f7a3e72b640 1 -- 172.21.15.119:0/3053614359 shutdown_connections 2024-03-29T11:19:26.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.357+0000 7f7a3e72b640 1 -- 172.21.15.119:0/3053614359 >> 172.21.15.119:0/3053614359 conn(0x7f7a38066b00 msgr2=0x7f7a38066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:26.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.357+0000 7f7a3e72b640 1 -- 172.21.15.119:0/3053614359 shutdown_connections 2024-03-29T11:19:26.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.357+0000 7f7a3e72b640 1 -- 172.21.15.119:0/3053614359 wait complete. 2024-03-29T11:19:26.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.357+0000 7f7a3e72b640 10 monclient: build_initial_monmap 2024-03-29T11:19:26.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.357+0000 7f7a3e72b640 10 monclient: monmap: 2024-03-29T11:19:26.349 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 0 2024-03-29T11:19:26.349 INFO:teuthology.orchestra.run.smithi119.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:26.349 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 0.000000 2024-03-29T11:19:26.349 INFO:teuthology.orchestra.run.smithi119.stderr:created 0.000000 2024-03-29T11:19:26.349 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:26.349 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:26.349 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:19:26.349 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:19:26.349 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:19:26.349 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:26.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.358+0000 7f7a3e72b640 1 librados: starting msgr at 2024-03-29T11:19:26.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.358+0000 7f7a3e72b640 1 librados: starting objecter 2024-03-29T11:19:26.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.358+0000 7f7a3e72b640 1 Processor -- start 2024-03-29T11:19:26.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.358+0000 7f7a3e72b640 1 -- start start 2024-03-29T11:19:26.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.358+0000 7f7a3e72b640 1 librados: setting wanted keys 2024-03-29T11:19:26.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.358+0000 7f7a3e72b640 1 librados: calling monclient init 2024-03-29T11:19:26.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.358+0000 7f7a3e72b640 10 monclient: init 2024-03-29T11:19:26.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.358+0000 7f7a3e72b640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:26.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.358+0000 7f7a3e72b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:26.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a3e72b640 10 monclient(hunting): picked mon.noname-b con 0x7f7a3809ac40 addr v1:172.21.15.119:6789/0 2024-03-29T11:19:26.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a3e72b640 10 monclient(hunting): picked mon.noname-c con 0x7f7a381406b0 addr v1:172.21.15.104:6790/0 2024-03-29T11:19:26.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a3e72b640 10 monclient(hunting): picked mon.noname-a con 0x7f7a380b07d0 addr v1:172.21.15.104:6789/0 2024-03-29T11:19:26.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a3e72b640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a38142220 con 0x7f7a380b07d0 2024-03-29T11:19:26.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a3e72b640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a381399e0 con 0x7f7a3809ac40 2024-03-29T11:19:26.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a3e72b640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a38143990 con 0x7f7a381406b0 2024-03-29T11:19:26.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a3e72b640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:26.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a3e72b640 10 monclient(hunting): authenticate will time out at 943.636084s 2024-03-29T11:19:26.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a37fff640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f7a3809ac40 0x7f7a3813ffa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:48430/0 (socket says 172.21.15.119:48430) 2024-03-29T11:19:26.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a37fff640 1 -- 172.21.15.119:0/649265074 learned_addr learned my addr 172.21.15.119:0/649265074 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-03-29T11:19:26.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 10 client.?.objecter ms_handle_connect 0x7f7a3809ac40 2024-03-29T11:19:26.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:26.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 1 -- 172.21.15.119:0/649265074 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3252175687 0 0) 0x7f7a381399e0 con 0x7f7a3809ac40 2024-03-29T11:19:26.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:19:26.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:26.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:26.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 10 monclient(hunting): my global_id is 4141 2024-03-29T11:19:26.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 1 -- 172.21.15.119:0/649265074 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a24004eb0 con 0x7f7a3809ac40 2024-03-29T11:19:26.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 10 client.?.objecter ms_handle_connect 0x7f7a380b07d0 2024-03-29T11:19:26.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:26.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 10 client.?.objecter ms_handle_connect 0x7f7a381406b0 2024-03-29T11:19:26.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:26.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 1 -- 172.21.15.119:0/649265074 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 678884390 0 0) 0x7f7a38142220 con 0x7f7a380b07d0 2024-03-29T11:19:26.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:19:26.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:26.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:26.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 10 monclient(hunting): my global_id is 4155 2024-03-29T11:19:26.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 1 -- 172.21.15.119:0/649265074 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a381399e0 con 0x7f7a380b07d0 2024-03-29T11:19:26.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 1 -- 172.21.15.119:0/649265074 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2587363803 0 0) 0x7f7a38143990 con 0x7f7a381406b0 2024-03-29T11:19:26.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:19:26.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:26.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:26.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 10 monclient(hunting): my global_id is 4151 2024-03-29T11:19:26.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 1 -- 172.21.15.119:0/649265074 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a38142220 con 0x7f7a381406b0 2024-03-29T11:19:26.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 1 -- 172.21.15.119:0/649265074 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2532611561 0 0) 0x7f7a24004eb0 con 0x7f7a3809ac40 2024-03-29T11:19:26.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 1 -- 172.21.15.119:0/649265074 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a38143990 con 0x7f7a3809ac40 2024-03-29T11:19:26.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 1 -- 172.21.15.119:0/649265074 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f7a2c001930 con 0x7f7a3809ac40 2024-03-29T11:19:26.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:26.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:26.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 10 monclient(hunting): dump: 2024-03-29T11:19:26.353 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 1 2024-03-29T11:19:26.353 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:26.353 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:26.353 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:26.353 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:26.353 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:26.353 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:26.353 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:26.353 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:26.353 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:26.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:26.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 1 -- 172.21.15.119:0/649265074 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3032869794 0 0) 0x7f7a381399e0 con 0x7f7a380b07d0 2024-03-29T11:19:26.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 1 -- 172.21.15.119:0/649265074 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a24004eb0 con 0x7f7a380b07d0 2024-03-29T11:19:26.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 1 -- 172.21.15.119:0/649265074 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f7a30001f00 con 0x7f7a380b07d0 2024-03-29T11:19:26.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:26.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:26.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 10 monclient(hunting): dump: 2024-03-29T11:19:26.354 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 1 2024-03-29T11:19:26.354 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:26.354 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:26.354 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:26.354 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:26.354 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:26.354 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:26.354 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:26.354 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:26.354 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:26.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 1 -- 172.21.15.119:0/649265074 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 340848641 0 0) 0x7f7a38142220 con 0x7f7a381406b0 2024-03-29T11:19:26.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 1 -- 172.21.15.119:0/649265074 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a381399e0 con 0x7f7a381406b0 2024-03-29T11:19:26.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 1 -- 172.21.15.119:0/649265074 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f7a28001920 con 0x7f7a381406b0 2024-03-29T11:19:26.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:26.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:26.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 10 monclient(hunting): dump: 2024-03-29T11:19:26.355 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 1 2024-03-29T11:19:26.355 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:26.355 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:26.355 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:26.355 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:26.355 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:26.355 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:26.355 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:26.355 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:26.355 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:26.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 1 -- 172.21.15.119:0/649265074 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2502415128 0 0) 0x7f7a38143990 con 0x7f7a3809ac40 2024-03-29T11:19:26.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 1 -- 172.21.15.119:0/649265074 >> v1:172.21.15.104:6790/0 conn(0x7f7a381406b0 legacy=0x7f7a38138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:26.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 1 -- 172.21.15.119:0/649265074 >> v1:172.21.15.104:6789/0 conn(0x7f7a380b07d0 legacy=0x7f7a381392d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:26.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:19:26.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 1 monclient: found mon.b 2024-03-29T11:19:26.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:26.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:19:26.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 1 -- 172.21.15.119:0/649265074 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a3813a1b0 con 0x7f7a3809ac40 2024-03-29T11:19:26.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 10 monclient: _finish_auth 0 2024-03-29T11:19:26.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:19:26.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:26.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a3e72b640 5 monclient: authenticate success, global_id 4141 2024-03-29T11:19:26.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a3e72b640 10 monclient: _renew_subs 2024-03-29T11:19:26.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a3e72b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:19:26.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a3e72b640 1 -- 172.21.15.119:0/649265074 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a38141330 con 0x7f7a3809ac40 2024-03-29T11:19:26.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a3e72b640 10 client.4141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:26.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a3e72b640 10 monclient: _renew_subs 2024-03-29T11:19:26.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a3e72b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:19:26.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a3e72b640 1 -- 172.21.15.119:0/649265074 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a381426b0 con 0x7f7a3809ac40 2024-03-29T11:19:26.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a3e72b640 1 librados: init done 2024-03-29T11:19:26.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 1 -- 172.21.15.119:0/649265074 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a2c003610 con 0x7f7a3809ac40 2024-03-29T11:19:26.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:26.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 1 -- 172.21.15.119:0/649265074 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f7a2c0039a0 con 0x7f7a3809ac40 2024-03-29T11:19:26.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:26.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:26.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a34ff9640 10 monclient: dump: 2024-03-29T11:19:26.357 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 1 2024-03-29T11:19:26.357 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:26.357 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:26.357 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:26.357 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:26.357 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:26.357 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:26.357 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:26.357 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:26.357 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:26.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.359+0000 7f7a367fc640 4 set_mon_vals no callback set 2024-03-29T11:19:26.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.360+0000 7f7a34ff9640 1 -- 172.21.15.119:0/649265074 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f7a2c003e90 con 0x7f7a3809ac40 2024-03-29T11:19:26.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.360+0000 7f7a34ff9640 1 -- 172.21.15.119:0/649265074 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 3588809544 0 0) 0x7f7a2c0067a0 con 0x7f7a3809ac40 2024-03-29T11:19:26.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.360+0000 7f7a34ff9640 10 client.4141.objecter ms_dispatch 0x7f7a38000b90 osd_map(8..8 src has 1..8) v4 2024-03-29T11:19:26.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.360+0000 7f7a34ff9640 3 client.4141.objecter handle_osd_map got epochs [8,8] > 0 2024-03-29T11:19:26.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.360+0000 7f7a34ff9640 3 client.4141.objecter handle_osd_map decoding full epoch 8 2024-03-29T11:19:26.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.360+0000 7f7a34ff9640 20 client.4141.objecter dump_active .. 0 homeless 2024-03-29T11:19:26.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.360+0000 7f7a3e72b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:26.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.360+0000 7f7a3e72b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:26.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.360+0000 7f7a3e72b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:19:26.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.360+0000 7f7a3e72b640 1 -- 172.21.15.119:0/649265074 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a3809b020 con 0x7f7a3809ac40 2024-03-29T11:19:26.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.364+0000 7f7a34ff9640 1 -- 172.21.15.119:0/649265074 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f7a2c004370 con 0x7f7a3809ac40 2024-03-29T11:19:26.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.364+0000 7f7a34ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:26.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.364+0000 7f7a34ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:26.372 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-29T11:19:26.395+0000 7f0ede043040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:19:26.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.436+0000 7f7a3e72b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "9c0a3b9b-48bb-4cd2-a973-ab112c32f77f", "id": 6}] 2024-03-29T11:19:26.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.436+0000 7f7a3e72b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "9c0a3b9b-48bb-4cd2-a973-ab112c32f77f", "id": 6}] 2024-03-29T11:19:26.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.436+0000 7f7a3e72b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:19:26.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.436+0000 7f7a3e72b640 1 -- 172.21.15.119:0/649265074 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "osd new", "uuid": "9c0a3b9b-48bb-4cd2-a973-ab112c32f77f", "id": 6} v 0) v1 -- 0x7f7a3804f730 con 0x7f7a3809ac40 2024-03-29T11:19:26.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.441+0000 7f7a34ff9640 1 -- 172.21.15.119:0/649265074 <== mon.1 v1:172.21.15.119:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9c0a3b9b-48bb-4cd2-a973-ab112c32f77f", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 1054482667 0 1113811849) 0x7f7a2c006a30 con 0x7f7a3809ac40 2024-03-29T11:19:26.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.441+0000 7f7a34ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "9c0a3b9b-48bb-4cd2-a973-ab112c32f77f", "id": 6}] 2024-03-29T11:19:26.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.441+0000 7f7a34ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:26.418 INFO:teuthology.orchestra.run.smithi119.stdout:6 2024-03-29T11:19:26.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.441+0000 7f7a3e72b640 10 librados: watch_flush enter 2024-03-29T11:19:26.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.441+0000 7f7a3e72b640 10 librados: watch_flush exit 2024-03-29T11:19:26.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.441+0000 7f7a3e72b640 20 client.4141.objecter shutdown clearing up homeless session... 2024-03-29T11:19:26.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.441+0000 7f7a3e72b640 10 client.4141.objecter successfully canceled tick 2024-03-29T11:19:26.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.441+0000 7f7a3e72b640 10 monclient: shutdown 2024-03-29T11:19:26.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.441+0000 7f7a3e72b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:26.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.441+0000 7f7a3e72b640 1 -- 172.21.15.119:0/649265074 >> v1:172.21.15.119:6789/0 conn(0x7f7a3809ac40 legacy=0x7f7a3813ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:26.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.442+0000 7f7a3e72b640 1 -- 172.21.15.119:0/649265074 shutdown_connections 2024-03-29T11:19:26.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.442+0000 7f7a3e72b640 1 -- 172.21.15.119:0/649265074 >> 172.21.15.119:0/649265074 conn(0x7f7a38066b00 msgr2=0x7f7a3813b2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:26.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.442+0000 7f7a3e72b640 1 -- 172.21.15.119:0/649265074 shutdown_connections 2024-03-29T11:19:26.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.442+0000 7f7a3e72b640 1 -- 172.21.15.119:0/649265074 wait complete. 2024-03-29T11:19:26.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.442+0000 7f7a3e72b640 1 librados: shutdown 2024-03-29T11:19:26.429 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster ceph osd new 352d1ad9-6778-43f4-a7e4-5345e7c19de0 7 2024-03-29T11:19:26.441 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-03-29T11:19:26.465+0000 7f32f3443040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:19:26.506 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-29T11:19:26.530+0000 7f0ede043040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:19:26.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.552+0000 7f70f67d6640 10 monclient: get_monmap_and_config 2024-03-29T11:19:26.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.552+0000 7f70f67d6640 10 monclient: build_initial_monmap 2024-03-29T11:19:26.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.552+0000 7f70f67d6640 10 monclient: monmap: 2024-03-29T11:19:26.529 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 0 2024-03-29T11:19:26.529 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:26.529 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:19:26.553315+0000 2024-03-29T11:19:26.529 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:19:26.553315+0000 2024-03-29T11:19:26.529 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:26.529 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:26.529 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:19:26.529 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:19:26.529 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:19:26.529 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:26.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.552+0000 7f70f67d6640 1 Processor -- start 2024-03-29T11:19:26.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.552+0000 7f70f67d6640 1 -- start start 2024-03-29T11:19:26.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.552+0000 7f70f67d6640 10 monclient: init 2024-03-29T11:19:26.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.553+0000 7f70f67d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:26.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.553+0000 7f70f67d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:26.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.553+0000 7f70f67d6640 10 monclient(hunting): picked mon.noname-a con 0x7f70f009ac40 addr v1:172.21.15.104:6789/0 2024-03-29T11:19:26.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.553+0000 7f70f67d6640 10 monclient(hunting): picked mon.noname-b con 0x7f70f009c870 addr v1:172.21.15.104:6790/0 2024-03-29T11:19:26.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.553+0000 7f70f67d6640 10 monclient(hunting): picked mon.noname-c con 0x7f70f00b4360 addr v1:172.21.15.119:6789/0 2024-03-29T11:19:26.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.553+0000 7f70f67d6640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70f009dec0 con 0x7f70f009ac40 2024-03-29T11:19:26.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.553+0000 7f70f67d6640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70f009da30 con 0x7f70f00b4360 2024-03-29T11:19:26.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.553+0000 7f70f67d6640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70f00b39c0 con 0x7f70f009c870 2024-03-29T11:19:26.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.553+0000 7f70f67d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:26.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.553+0000 7f70f67d6640 10 monclient(hunting): authenticate will time out at 943.830489s 2024-03-29T11:19:26.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.553+0000 7f70f4d73640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f70f00b4360 0x7f70f014d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:48438/0 (socket says 172.21.15.119:48438) 2024-03-29T11:19:26.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.553+0000 7f70f4d73640 1 -- 172.21.15.119:0/1583499921 learned_addr learned my addr 172.21.15.119:0/1583499921 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-03-29T11:19:26.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.553+0000 7f70eeffd640 1 -- 172.21.15.119:0/1583499921 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1139801443 0 0) 0x7f70f009da30 con 0x7f70f00b4360 2024-03-29T11:19:26.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.553+0000 7f70eeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:26.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.553+0000 7f70eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:26.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.553+0000 7f70eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:26.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.553+0000 7f70eeffd640 10 monclient(hunting): my global_id is 4144 2024-03-29T11:19:26.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.553+0000 7f70eeffd640 1 -- 172.21.15.119:0/1583499921 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70d8003520 con 0x7f70f00b4360 2024-03-29T11:19:26.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.553+0000 7f70eeffd640 1 -- 172.21.15.119:0/1583499921 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2325483197 0 0) 0x7f70e4001110 con 0x7f70f009ac40 2024-03-29T11:19:26.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.553+0000 7f70eeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:26.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.553+0000 7f70eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:26.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.553+0000 7f70eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:26.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.553+0000 7f70eeffd640 10 monclient(hunting): my global_id is 4158 2024-03-29T11:19:26.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.553+0000 7f70eeffd640 1 -- 172.21.15.119:0/1583499921 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70f009da30 con 0x7f70f009ac40 2024-03-29T11:19:26.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.553+0000 7f70eeffd640 1 -- 172.21.15.119:0/1583499921 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3660240483 0 0) 0x7f70f00b39c0 con 0x7f70f009c870 2024-03-29T11:19:26.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.553+0000 7f70eeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:26.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.553+0000 7f70eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:26.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.553+0000 7f70eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:26.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.553+0000 7f70eeffd640 10 monclient(hunting): my global_id is 4154 2024-03-29T11:19:26.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.553+0000 7f70eeffd640 1 -- 172.21.15.119:0/1583499921 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70e4001110 con 0x7f70f009c870 2024-03-29T11:19:26.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.554+0000 7f70eeffd640 1 -- 172.21.15.119:0/1583499921 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1288263107 0 0) 0x7f70d8003520 con 0x7f70f00b4360 2024-03-29T11:19:26.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.554+0000 7f70eeffd640 1 -- 172.21.15.119:0/1583499921 >> v1:172.21.15.104:6790/0 conn(0x7f70f009c870 legacy=0x7f70f013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:26.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.554+0000 7f70eeffd640 1 -- 172.21.15.119:0/1583499921 >> v1:172.21.15.104:6789/0 conn(0x7f70f009ac40 legacy=0x7f70f00b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:26.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.554+0000 7f70eeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:26.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.554+0000 7f70eeffd640 1 monclient: found mon.noname-c 2024-03-29T11:19:26.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.554+0000 7f70eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:26.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.554+0000 7f70eeffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.119:6789/0 2024-03-29T11:19:26.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.554+0000 7f70eeffd640 1 -- 172.21.15.119:0/1583499921 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70f00b5160 con 0x7f70f00b4360 2024-03-29T11:19:26.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.554+0000 7f70eeffd640 10 monclient: _finish_auth 0 2024-03-29T11:19:26.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.554+0000 7f70eeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:26.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.554+0000 7f70eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:26.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.554+0000 7f70eeffd640 1 -- 172.21.15.119:0/1583499921 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f70e8001a10 con 0x7f70f00b4360 2024-03-29T11:19:26.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.554+0000 7f70f67d6640 5 monclient: authenticate success, global_id 4144 2024-03-29T11:19:26.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.554+0000 7f70f67d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:26.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.554+0000 7f70eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:26.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.554+0000 7f70eeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:26.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.554+0000 7f70eeffd640 10 monclient: dump: 2024-03-29T11:19:26.542 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 1 2024-03-29T11:19:26.542 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:26.542 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:26.542 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:26.542 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:26.542 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:26.542 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:26.542 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:26.542 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:26.542 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:26.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.554+0000 7f70f67d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:26.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.554+0000 7f70eeffd640 1 -- 172.21.15.119:0/1583499921 <== mon.1 v1:172.21.15.119:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f70e8001bf0 con 0x7f70f00b4360 2024-03-29T11:19:26.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.554+0000 7f70eeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:26.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.554+0000 7f70eeffd640 1 -- 172.21.15.119:0/1583499921 <== mon.1 v1:172.21.15.119:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f70e8002000 con 0x7f70f00b4360 2024-03-29T11:19:26.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.554+0000 7f70eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:26.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.554+0000 7f70eeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:26.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.554+0000 7f70eeffd640 10 monclient: dump: 2024-03-29T11:19:26.543 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 1 2024-03-29T11:19:26.543 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:26.543 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:26.543 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:26.543 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:26.543 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:26.543 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:26.543 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:26.543 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:26.543 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:26.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.554+0000 7f70f67d6640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:26.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.554+0000 7f70f67d6640 4 set_mon_vals no callback set 2024-03-29T11:19:26.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.554+0000 7f70f67d6640 10 monclient: shutdown 2024-03-29T11:19:26.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.554+0000 7f70f67d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:26.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.554+0000 7f70f67d6640 1 -- 172.21.15.119:0/1583499921 >> v1:172.21.15.119:6789/0 conn(0x7f70f00b4360 legacy=0x7f70f014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:26.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.555+0000 7f70f67d6640 1 -- 172.21.15.119:0/1583499921 shutdown_connections 2024-03-29T11:19:26.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.555+0000 7f70f67d6640 1 -- 172.21.15.119:0/1583499921 >> 172.21.15.119:0/1583499921 conn(0x7f70f0066b00 msgr2=0x7f70f0066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:26.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.555+0000 7f70f67d6640 1 -- 172.21.15.119:0/1583499921 shutdown_connections 2024-03-29T11:19:26.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.555+0000 7f70f67d6640 1 -- 172.21.15.119:0/1583499921 wait complete. 2024-03-29T11:19:26.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.555+0000 7f70f67d6640 10 monclient: build_initial_monmap 2024-03-29T11:19:26.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.555+0000 7f70f67d6640 10 monclient: monmap: 2024-03-29T11:19:26.544 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 0 2024-03-29T11:19:26.544 INFO:teuthology.orchestra.run.smithi119.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:26.544 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 0.000000 2024-03-29T11:19:26.544 INFO:teuthology.orchestra.run.smithi119.stderr:created 0.000000 2024-03-29T11:19:26.544 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:26.544 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:26.545 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:19:26.545 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:19:26.545 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:19:26.545 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:26.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.555+0000 7f70f67d6640 1 librados: starting msgr at 2024-03-29T11:19:26.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.555+0000 7f70f67d6640 1 librados: starting objecter 2024-03-29T11:19:26.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.556+0000 7f70f67d6640 1 Processor -- start 2024-03-29T11:19:26.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.556+0000 7f70f67d6640 1 -- start start 2024-03-29T11:19:26.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.556+0000 7f70f67d6640 1 librados: setting wanted keys 2024-03-29T11:19:26.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.556+0000 7f70f67d6640 1 librados: calling monclient init 2024-03-29T11:19:26.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.556+0000 7f70f67d6640 10 monclient: init 2024-03-29T11:19:26.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.556+0000 7f70f67d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:26.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.556+0000 7f70f67d6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:26.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.556+0000 7f70f67d6640 10 monclient(hunting): picked mon.noname-b con 0x7f70f009ac40 addr v1:172.21.15.119:6789/0 2024-03-29T11:19:26.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.556+0000 7f70f67d6640 10 monclient(hunting): picked mon.noname-a con 0x7f70f0141910 addr v1:172.21.15.104:6789/0 2024-03-29T11:19:26.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.556+0000 7f70f67d6640 10 monclient(hunting): picked mon.noname-c con 0x7f70f013ffa0 addr v1:172.21.15.104:6790/0 2024-03-29T11:19:26.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.556+0000 7f70f67d6640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70f0139b60 con 0x7f70f0141910 2024-03-29T11:19:26.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.556+0000 7f70f67d6640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70f01d6f10 con 0x7f70f009ac40 2024-03-29T11:19:26.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.556+0000 7f70f67d6640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70f00afba0 con 0x7f70f013ffa0 2024-03-29T11:19:26.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.556+0000 7f70f67d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:26.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.556+0000 7f70f67d6640 10 monclient(hunting): authenticate will time out at 943.833809s 2024-03-29T11:19:26.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.556+0000 7f70effff640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f70f009ac40 0x7f70f0141200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:48446/0 (socket says 172.21.15.119:48446) 2024-03-29T11:19:26.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.556+0000 7f70effff640 1 -- 172.21.15.119:0/1532825310 learned_addr learned my addr 172.21.15.119:0/1532825310 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-03-29T11:19:26.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.556+0000 7f70ef7fe640 1 --1- 172.21.15.119:0/1532825310 >> v1:172.21.15.104:6789/0 conn(0x7f70f0141910 0x7f70f01d73a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.119:42710/0 (socket says 172.21.15.119:42710) 2024-03-29T11:19:26.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.556+0000 7f70ecff9640 10 client.?.objecter ms_handle_connect 0x7f70f009ac40 2024-03-29T11:19:26.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.556+0000 7f70ecff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:26.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.556+0000 7f70ecff9640 10 client.?.objecter ms_handle_connect 0x7f70f013ffa0 2024-03-29T11:19:26.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.556+0000 7f70ecff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:26.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.556+0000 7f70ecff9640 10 client.?.objecter ms_handle_connect 0x7f70f0141910 2024-03-29T11:19:26.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.556+0000 7f70ecff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:26.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 1 -- 172.21.15.119:0/1532825310 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 402839237 0 0) 0x7f70f01d6f10 con 0x7f70f009ac40 2024-03-29T11:19:26.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 10 monclient(hunting): _negotiate 2024-03-29T11:19:26.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:26.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:26.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 10 monclient(hunting): my global_id is 4147 2024-03-29T11:19:26.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 1 -- 172.21.15.119:0/1532825310 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70dc004eb0 con 0x7f70f009ac40 2024-03-29T11:19:26.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 1 -- 172.21.15.119:0/1532825310 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3434239990 0 0) 0x7f70f00afba0 con 0x7f70f013ffa0 2024-03-29T11:19:26.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 10 monclient(hunting): _negotiate 2024-03-29T11:19:26.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:26.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:26.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 10 monclient(hunting): my global_id is 4157 2024-03-29T11:19:26.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 1 -- 172.21.15.119:0/1532825310 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70f01d6f10 con 0x7f70f013ffa0 2024-03-29T11:19:26.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 1 -- 172.21.15.119:0/1532825310 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 827860556 0 0) 0x7f70e4001110 con 0x7f70f0141910 2024-03-29T11:19:26.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 10 monclient(hunting): _negotiate 2024-03-29T11:19:26.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:26.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:26.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 10 monclient(hunting): my global_id is 4161 2024-03-29T11:19:26.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 1 -- 172.21.15.119:0/1532825310 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70f00afba0 con 0x7f70f0141910 2024-03-29T11:19:26.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 1 -- 172.21.15.119:0/1532825310 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1576769574 0 0) 0x7f70dc004eb0 con 0x7f70f009ac40 2024-03-29T11:19:26.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 1 -- 172.21.15.119:0/1532825310 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70e4001110 con 0x7f70f009ac40 2024-03-29T11:19:26.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 1 -- 172.21.15.119:0/1532825310 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f70e4001a20 con 0x7f70f009ac40 2024-03-29T11:19:26.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:26.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:26.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 10 monclient(hunting): dump: 2024-03-29T11:19:26.548 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 1 2024-03-29T11:19:26.548 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:26.548 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:26.548 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:26.549 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:26.549 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:26.549 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:26.549 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:26.549 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:26.549 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:26.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:26.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 1 -- 172.21.15.119:0/1532825310 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1513663453 0 0) 0x7f70f00afba0 con 0x7f70f0141910 2024-03-29T11:19:26.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 1 -- 172.21.15.119:0/1532825310 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70dc004eb0 con 0x7f70f0141910 2024-03-29T11:19:26.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 1 -- 172.21.15.119:0/1532825310 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3089272052 0 0) 0x7f70f01d6f10 con 0x7f70f013ffa0 2024-03-29T11:19:26.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 1 -- 172.21.15.119:0/1532825310 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70f00afba0 con 0x7f70f013ffa0 2024-03-29T11:19:26.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 1 -- 172.21.15.119:0/1532825310 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f70e0001650 con 0x7f70f0141910 2024-03-29T11:19:26.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:26.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:26.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 10 monclient(hunting): dump: 2024-03-29T11:19:26.550 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 1 2024-03-29T11:19:26.550 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:26.550 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:26.550 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:26.550 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:26.550 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:26.550 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:26.550 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:26.550 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:26.550 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:26.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 1 -- 172.21.15.119:0/1532825310 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f70e8001f00 con 0x7f70f013ffa0 2024-03-29T11:19:26.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:26.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:26.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 10 monclient(hunting): dump: 2024-03-29T11:19:26.550 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 1 2024-03-29T11:19:26.551 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:26.551 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:26.551 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:26.551 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:26.551 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:26.551 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:26.551 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:26.551 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:26.551 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:26.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 1 -- 172.21.15.119:0/1532825310 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2269474762 0 0) 0x7f70e4001110 con 0x7f70f009ac40 2024-03-29T11:19:26.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 1 -- 172.21.15.119:0/1532825310 >> v1:172.21.15.104:6790/0 conn(0x7f70f013ffa0 legacy=0x7f70f0138e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:26.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 1 -- 172.21.15.119:0/1532825310 >> v1:172.21.15.104:6789/0 conn(0x7f70f0141910 legacy=0x7f70f01d73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:26.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 10 monclient: _finish_hunting 0 2024-03-29T11:19:26.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 1 monclient: found mon.b 2024-03-29T11:19:26.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:26.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:19:26.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 1 -- 172.21.15.119:0/1532825310 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70f0139520 con 0x7f70f009ac40 2024-03-29T11:19:26.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 10 monclient: _finish_auth 0 2024-03-29T11:19:26.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 10 monclient: _check_auth_tickets 2024-03-29T11:19:26.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:26.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70f67d6640 5 monclient: authenticate success, global_id 4147 2024-03-29T11:19:26.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70f67d6640 10 monclient: _renew_subs 2024-03-29T11:19:26.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70f67d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:19:26.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70f67d6640 1 -- 172.21.15.119:0/1532825310 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70f0139710 con 0x7f70f009ac40 2024-03-29T11:19:26.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70f67d6640 10 client.4147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:26.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70f67d6640 10 monclient: _renew_subs 2024-03-29T11:19:26.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70f67d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:19:26.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70f67d6640 1 -- 172.21.15.119:0/1532825310 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f70f01d7c90 con 0x7f70f009ac40 2024-03-29T11:19:26.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70f67d6640 1 librados: init done 2024-03-29T11:19:26.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 1 -- 172.21.15.119:0/1532825310 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f70e4001c80 con 0x7f70f009ac40 2024-03-29T11:19:26.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:26.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 1 -- 172.21.15.119:0/1532825310 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f70e4003a60 con 0x7f70f009ac40 2024-03-29T11:19:26.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:26.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:26.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ee7fc640 4 set_mon_vals no callback set 2024-03-29T11:19:26.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 10 monclient: dump: 2024-03-29T11:19:26.553 INFO:teuthology.orchestra.run.smithi119.stderr:epoch 1 2024-03-29T11:19:26.553 INFO:teuthology.orchestra.run.smithi119.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:26.553 INFO:teuthology.orchestra.run.smithi119.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:26.553 INFO:teuthology.orchestra.run.smithi119.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:26.553 INFO:teuthology.orchestra.run.smithi119.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:26.553 INFO:teuthology.orchestra.run.smithi119.stderr:election_strategy: 1 2024-03-29T11:19:26.553 INFO:teuthology.orchestra.run.smithi119.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:26.553 INFO:teuthology.orchestra.run.smithi119.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:26.554 INFO:teuthology.orchestra.run.smithi119.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:26.554 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:19:26.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 1 -- 172.21.15.119:0/1532825310 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f70e4004010 con 0x7f70f009ac40 2024-03-29T11:19:26.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 1 -- 172.21.15.119:0/1532825310 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 2568585685 0 0) 0x7f70e40069a0 con 0x7f70f009ac40 2024-03-29T11:19:26.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 10 client.4147.objecter ms_dispatch 0x7f70f0000b90 osd_map(9..9 src has 1..9) v4 2024-03-29T11:19:26.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 3 client.4147.objecter handle_osd_map got epochs [9,9] > 0 2024-03-29T11:19:26.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 3 client.4147.objecter handle_osd_map decoding full epoch 9 2024-03-29T11:19:26.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.557+0000 7f70ecff9640 20 client.4147.objecter dump_active .. 0 homeless 2024-03-29T11:19:26.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.558+0000 7f70f67d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:26.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.558+0000 7f70f67d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:26.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.558+0000 7f70f67d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:19:26.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.558+0000 7f70f67d6640 1 -- 172.21.15.119:0/1532825310 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70d0005080 con 0x7f70f009ac40 2024-03-29T11:19:26.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.560+0000 7f70ecff9640 1 -- 172.21.15.119:0/1532825310 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f70e4006bb0 con 0x7f70f009ac40 2024-03-29T11:19:26.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.560+0000 7f70ecff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:26.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.560+0000 7f70ecff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:26.583 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-03-29T11:19:26.606+0000 7f32f3443040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:19:26.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.633+0000 7f70f67d6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "352d1ad9-6778-43f4-a7e4-5345e7c19de0", "id": 7}] 2024-03-29T11:19:26.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.633+0000 7f70f67d6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "352d1ad9-6778-43f4-a7e4-5345e7c19de0", "id": 7}] 2024-03-29T11:19:26.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.633+0000 7f70f67d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:19:26.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.633+0000 7f70f67d6640 1 -- 172.21.15.119:0/1532825310 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "osd new", "uuid": "352d1ad9-6778-43f4-a7e4-5345e7c19de0", "id": 7} v 0) v1 -- 0x7f70d00058b0 con 0x7f70f009ac40 2024-03-29T11:19:26.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.636+0000 7f70ecff9640 1 -- 172.21.15.119:0/1532825310 <== mon.1 v1:172.21.15.119:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "352d1ad9-6778-43f4-a7e4-5345e7c19de0", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 3864378778 0 1371665406) 0x7f70e40062f0 con 0x7f70f009ac40 2024-03-29T11:19:26.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.637+0000 7f70ecff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "352d1ad9-6778-43f4-a7e4-5345e7c19de0", "id": 7}] 2024-03-29T11:19:26.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.637+0000 7f70ecff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:26.614 INFO:teuthology.orchestra.run.smithi119.stdout:7 2024-03-29T11:19:26.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.637+0000 7f70f67d6640 10 librados: watch_flush enter 2024-03-29T11:19:26.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.637+0000 7f70f67d6640 10 librados: watch_flush exit 2024-03-29T11:19:26.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.637+0000 7f70f67d6640 20 client.4147.objecter shutdown clearing up homeless session... 2024-03-29T11:19:26.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.637+0000 7f70f67d6640 10 client.4147.objecter successfully canceled tick 2024-03-29T11:19:26.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.637+0000 7f70f67d6640 10 monclient: shutdown 2024-03-29T11:19:26.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.637+0000 7f70f67d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:26.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.637+0000 7f70f67d6640 1 -- 172.21.15.119:0/1532825310 >> v1:172.21.15.119:6789/0 conn(0x7f70f009ac40 legacy=0x7f70f0141200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:26.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.637+0000 7f70f67d6640 1 -- 172.21.15.119:0/1532825310 shutdown_connections 2024-03-29T11:19:26.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.637+0000 7f70f67d6640 1 -- 172.21.15.119:0/1532825310 >> 172.21.15.119:0/1532825310 conn(0x7f70f0066b00 msgr2=0x7f70f013c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:26.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.638+0000 7f70f67d6640 1 -- 172.21.15.119:0/1532825310 shutdown_connections 2024-03-29T11:19:26.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.638+0000 7f70f67d6640 1 -- 172.21.15.119:0/1532825310 wait complete. 2024-03-29T11:19:26.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-03-29T11:19:26.638+0000 7f70f67d6640 1 librados: shutdown 2024-03-29T11:19:26.625 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T11:19:26.625 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-29T11:19:26.627 INFO:tasks.ceph.osd.0:Started 2024-03-29T11:19:26.627 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T11:19:26.627 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-29T11:19:26.629 INFO:tasks.ceph.osd.1:Started 2024-03-29T11:19:26.629 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T11:19:26.629 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-29T11:19:26.631 INFO:tasks.ceph.osd.2:Started 2024-03-29T11:19:26.631 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-29T11:19:26.631 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-29T11:19:26.633 INFO:tasks.ceph.osd.3:Started 2024-03-29T11:19:26.634 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-29T11:19:26.634 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-29T11:19:26.635 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-29T11:19:26.658+0000 7f0ede043040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:19:26.667 INFO:tasks.ceph.osd.4:Started 2024-03-29T11:19:26.667 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-29T11:19:26.667 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-29T11:19:26.669 INFO:tasks.ceph.osd.5:Started 2024-03-29T11:19:26.669 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-29T11:19:26.669 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-29T11:19:26.672 INFO:tasks.ceph.osd.6:Started 2024-03-29T11:19:26.672 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-29T11:19:26.672 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-29T11:19:26.674 INFO:tasks.ceph.osd.7:Started 2024-03-29T11:19:26.674 DEBUG:teuthology.orchestra.run.smithi104:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:26.696 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-29T11:19:26.719+0000 7f0ede043040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:19:26.724 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-03-29T11:19:26.747+0000 7f32f3443040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:19:26.777 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-03-29T11:19:26.801+0000 7f32f3443040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:19:26.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.812+0000 7f0d1326e640 10 monclient: get_monmap_and_config 2024-03-29T11:19:26.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.812+0000 7f0d1326e640 10 monclient: build_initial_monmap 2024-03-29T11:19:26.789 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.812+0000 7f0d1326e640 10 monclient: monmap: 2024-03-29T11:19:26.789 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-03-29T11:19:26.789 INFO:teuthology.orchestra.run.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:26.789 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-03-29T11:19:26.813725+0000 2024-03-29T11:19:26.789 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-03-29T11:19:26.813725+0000 2024-03-29T11:19:26.789 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:26.790 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-03-29T11:19:26.790 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:19:26.790 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:19:26.790 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:19:26.790 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-03-29T11:19:26.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.813+0000 7f0d1326e640 1 Processor -- start 2024-03-29T11:19:26.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.813+0000 7f0d1326e640 1 -- start start 2024-03-29T11:19:26.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.813+0000 7f0d1326e640 10 monclient: init 2024-03-29T11:19:26.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.814+0000 7f0d1326e640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:26.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.814+0000 7f0d1326e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:26.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.814+0000 7f0d1326e640 10 monclient(hunting): picked mon.noname-c con 0x7f0d0c09ac40 addr v1:172.21.15.119:6789/0 2024-03-29T11:19:26.797 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.814+0000 7f0d1326e640 10 monclient(hunting): picked mon.noname-b con 0x7f0d0c09c870 addr v1:172.21.15.104:6790/0 2024-03-29T11:19:26.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.814+0000 7f0d1326e640 10 monclient(hunting): picked mon.noname-a con 0x7f0d0c0b4360 addr v1:172.21.15.104:6789/0 2024-03-29T11:19:26.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.814+0000 7f0d1326e640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d0c09dec0 con 0x7f0d0c0b4360 2024-03-29T11:19:26.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.814+0000 7f0d1326e640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d0c09da30 con 0x7f0d0c09ac40 2024-03-29T11:19:26.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.814+0000 7f0d1326e640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d0c0b39c0 con 0x7f0d0c09c870 2024-03-29T11:19:26.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.814+0000 7f0d1326e640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:26.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.814+0000 7f0d1326e640 10 monclient(hunting): authenticate will time out at 948.065255s 2024-03-29T11:19:26.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.814+0000 7f0d10809640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f0d0c09c870 0x7f0d0c13f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:40068/0 (socket says 172.21.15.104:40068) 2024-03-29T11:19:26.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.814+0000 7f0d10809640 1 -- 172.21.15.104:0/3788508517 learned_addr learned my addr 172.21.15.104:0/3788508517 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:19:26.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.814+0000 7f0d03fff640 1 -- 172.21.15.104:0/3788508517 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2233628890 0 0) 0x7f0d04001490 con 0x7f0d0c0b4360 2024-03-29T11:19:26.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.814+0000 7f0d03fff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:26.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.814+0000 7f0d03fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:26.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.814+0000 7f0d03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:26.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.814+0000 7f0d03fff640 10 monclient(hunting): my global_id is 4188 2024-03-29T11:19:26.798 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.814+0000 7f0d03fff640 1 -- 172.21.15.104:0/3788508517 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cf8003520 con 0x7f0d0c0b4360 2024-03-29T11:19:26.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.814+0000 7f0d03fff640 1 -- 172.21.15.104:0/3788508517 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1714927082 0 0) 0x7f0d0c0b39c0 con 0x7f0d0c09c870 2024-03-29T11:19:26.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.814+0000 7f0d03fff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:26.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.814+0000 7f0d03fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:26.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.814+0000 7f0d03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:26.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.814+0000 7f0d03fff640 10 monclient(hunting): my global_id is 4184 2024-03-29T11:19:26.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.814+0000 7f0d03fff640 1 -- 172.21.15.104:0/3788508517 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d04001490 con 0x7f0d0c09c870 2024-03-29T11:19:26.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.814+0000 7f0d03fff640 1 -- 172.21.15.104:0/3788508517 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 404408887 0 0) 0x7f0d0c09da30 con 0x7f0d0c09ac40 2024-03-29T11:19:26.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.814+0000 7f0d03fff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:26.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.814+0000 7f0d03fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:26.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.814+0000 7f0d03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:26.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.814+0000 7f0d03fff640 10 monclient(hunting): my global_id is 4174 2024-03-29T11:19:26.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.814+0000 7f0d03fff640 1 -- 172.21.15.104:0/3788508517 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d0c0b39c0 con 0x7f0d0c09ac40 2024-03-29T11:19:26.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.814+0000 7f0d03fff640 1 -- 172.21.15.104:0/3788508517 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2021825637 0 0) 0x7f0cf8003520 con 0x7f0d0c0b4360 2024-03-29T11:19:26.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.814+0000 7f0d03fff640 1 -- 172.21.15.104:0/3788508517 >> v1:172.21.15.104:6790/0 conn(0x7f0d0c09c870 legacy=0x7f0d0c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:26.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.814+0000 7f0d03fff640 1 -- 172.21.15.104:0/3788508517 >> v1:172.21.15.119:6789/0 conn(0x7f0d0c09ac40 legacy=0x7f0d0c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:26.799 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.814+0000 7f0d03fff640 10 monclient: _finish_hunting 0 2024-03-29T11:19:26.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.814+0000 7f0d03fff640 1 monclient: found mon.noname-a 2024-03-29T11:19:26.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.814+0000 7f0d03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:26.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.814+0000 7f0d03fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.104:6789/0 2024-03-29T11:19:26.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.814+0000 7f0d03fff640 1 -- 172.21.15.104:0/3788508517 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d0c0b5160 con 0x7f0d0c0b4360 2024-03-29T11:19:26.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.814+0000 7f0d03fff640 10 monclient: _finish_auth 0 2024-03-29T11:19:26.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.814+0000 7f0d03fff640 10 monclient: _check_auth_tickets 2024-03-29T11:19:26.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.814+0000 7f0d03fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:26.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.814+0000 7f0d1326e640 5 monclient: authenticate success, global_id 4188 2024-03-29T11:19:26.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.814+0000 7f0d03fff640 1 -- 172.21.15.104:0/3788508517 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f0d04001a00 con 0x7f0d0c0b4360 2024-03-29T11:19:26.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.814+0000 7f0d1326e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:26.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.814+0000 7f0d03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:26.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.814+0000 7f0d03fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:26.800 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.814+0000 7f0d03fff640 10 monclient: dump: 2024-03-29T11:19:26.800 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-03-29T11:19:26.800 INFO:teuthology.orchestra.run.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:26.800 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:26.800 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:26.801 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:26.801 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-03-29T11:19:26.801 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:26.801 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:26.801 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:26.801 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-03-29T11:19:26.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.814+0000 7f0d1326e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:26.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.814+0000 7f0d03fff640 1 -- 172.21.15.104:0/3788508517 <== mon.0 v1:172.21.15.104:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d04001b60 con 0x7f0d0c0b4360 2024-03-29T11:19:26.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.815+0000 7f0d03fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:26.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.815+0000 7f0d03fff640 1 -- 172.21.15.104:0/3788508517 <== mon.0 v1:172.21.15.104:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f0d04001fb0 con 0x7f0d0c0b4360 2024-03-29T11:19:26.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.815+0000 7f0d1326e640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:26.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.815+0000 7f0d1326e640 4 set_mon_vals no callback set 2024-03-29T11:19:26.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.815+0000 7f0d03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:26.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.815+0000 7f0d03fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:26.801 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.815+0000 7f0d03fff640 10 monclient: dump: 2024-03-29T11:19:26.801 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-03-29T11:19:26.801 INFO:teuthology.orchestra.run.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:26.802 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:26.802 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:26.802 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:26.802 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-03-29T11:19:26.802 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:26.802 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:26.802 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:26.802 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-03-29T11:19:26.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.815+0000 7f0d1326e640 10 monclient: shutdown 2024-03-29T11:19:26.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.815+0000 7f0d1326e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:26.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.815+0000 7f0d1326e640 1 -- 172.21.15.104:0/3788508517 >> v1:172.21.15.104:6789/0 conn(0x7f0d0c0b4360 legacy=0x7f0d0c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:26.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.815+0000 7f0d1326e640 1 -- 172.21.15.104:0/3788508517 shutdown_connections 2024-03-29T11:19:26.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.815+0000 7f0d1326e640 1 -- 172.21.15.104:0/3788508517 >> 172.21.15.104:0/3788508517 conn(0x7f0d0c066b00 msgr2=0x7f0d0c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:26.802 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.815+0000 7f0d1326e640 1 -- 172.21.15.104:0/3788508517 shutdown_connections 2024-03-29T11:19:26.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.815+0000 7f0d1326e640 1 -- 172.21.15.104:0/3788508517 wait complete. 2024-03-29T11:19:26.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.816+0000 7f0d1326e640 10 monclient: build_initial_monmap 2024-03-29T11:19:26.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.816+0000 7f0d1326e640 10 monclient: monmap: 2024-03-29T11:19:26.803 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 0 2024-03-29T11:19:26.803 INFO:teuthology.orchestra.run.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:26.803 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 0.000000 2024-03-29T11:19:26.803 INFO:teuthology.orchestra.run.smithi104.stderr:created 0.000000 2024-03-29T11:19:26.803 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:26.803 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-03-29T11:19:26.803 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:19:26.803 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:19:26.803 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:19:26.803 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-03-29T11:19:26.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.816+0000 7f0d1326e640 1 librados: starting msgr at 2024-03-29T11:19:26.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.816+0000 7f0d1326e640 1 librados: starting objecter 2024-03-29T11:19:26.803 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.816+0000 7f0d1326e640 1 Processor -- start 2024-03-29T11:19:26.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.816+0000 7f0d1326e640 1 -- start start 2024-03-29T11:19:26.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.816+0000 7f0d1326e640 1 librados: setting wanted keys 2024-03-29T11:19:26.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.816+0000 7f0d1326e640 1 librados: calling monclient init 2024-03-29T11:19:26.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.816+0000 7f0d1326e640 10 monclient: init 2024-03-29T11:19:26.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.817+0000 7f0d1326e640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:26.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.817+0000 7f0d1326e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:26.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.817+0000 7f0d1326e640 10 monclient(hunting): picked mon.noname-b con 0x7f0d0c09ac40 addr v1:172.21.15.119:6789/0 2024-03-29T11:19:26.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.817+0000 7f0d1326e640 10 monclient(hunting): picked mon.noname-a con 0x7f0d0c142380 addr v1:172.21.15.104:6789/0 2024-03-29T11:19:26.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.817+0000 7f0d1326e640 10 monclient(hunting): picked mon.noname-c con 0x7f0d0c1d7010 addr v1:172.21.15.104:6790/0 2024-03-29T11:19:26.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.817+0000 7f0d1326e640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d0c143000 con 0x7f0d0c142380 2024-03-29T11:19:26.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.817+0000 7f0d10809640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f0d0c142380 0x7f0d0c1389c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:42410/0 (socket says 172.21.15.104:42410) 2024-03-29T11:19:26.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.817+0000 7f0d10809640 1 -- 172.21.15.104:0/4115142224 learned_addr learned my addr 172.21.15.104:0/4115142224 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:19:26.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.817+0000 7f0d1326e640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d0c1397e0 con 0x7f0d0c09ac40 2024-03-29T11:19:26.804 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.817+0000 7f0d1326e640 1 -- 172.21.15.104:0/4115142224 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d0c0acd00 con 0x7f0d0c1d7010 2024-03-29T11:19:26.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.817+0000 7f0d1326e640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:26.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.817+0000 7f0d01ffb640 10 client.?.objecter ms_handle_connect 0x7f0d0c142380 2024-03-29T11:19:26.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.817+0000 7f0d01ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:26.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.817+0000 7f0d01ffb640 10 client.?.objecter ms_handle_connect 0x7f0d0c1d7010 2024-03-29T11:19:26.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.817+0000 7f0d01ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:26.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.817+0000 7f0d1326e640 10 monclient(hunting): authenticate will time out at 948.068614s 2024-03-29T11:19:26.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.817+0000 7f0d01ffb640 10 client.?.objecter ms_handle_connect 0x7f0d0c09ac40 2024-03-29T11:19:26.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.817+0000 7f0d01ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:26.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.817+0000 7f0d01ffb640 1 -- 172.21.15.104:0/4115142224 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2598601121 0 0) 0x7f0d0c1397e0 con 0x7f0d0c09ac40 2024-03-29T11:19:26.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.817+0000 7f0d01ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:26.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.817+0000 7f0d01ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:26.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.817+0000 7f0d01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:26.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.817+0000 7f0d01ffb640 10 monclient(hunting): my global_id is 4177 2024-03-29T11:19:26.805 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.817+0000 7f0d01ffb640 1 -- 172.21.15.104:0/4115142224 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cf0004eb0 con 0x7f0d0c09ac40 2024-03-29T11:19:26.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.817+0000 7f0d01ffb640 1 -- 172.21.15.104:0/4115142224 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3349604806 0 0) 0x7f0d0c143000 con 0x7f0d0c142380 2024-03-29T11:19:26.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.817+0000 7f0d01ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:26.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.817+0000 7f0d01ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:26.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.817+0000 7f0d01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:26.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.817+0000 7f0d01ffb640 10 monclient(hunting): my global_id is 4191 2024-03-29T11:19:26.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.817+0000 7f0d01ffb640 1 -- 172.21.15.104:0/4115142224 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d0c1397e0 con 0x7f0d0c142380 2024-03-29T11:19:26.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.817+0000 7f0d01ffb640 1 -- 172.21.15.104:0/4115142224 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3989301774 0 0) 0x7f0d0c0acd00 con 0x7f0d0c1d7010 2024-03-29T11:19:26.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.817+0000 7f0d01ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:26.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.817+0000 7f0d01ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:26.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.817+0000 7f0d01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:26.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.817+0000 7f0d01ffb640 10 monclient(hunting): my global_id is 4187 2024-03-29T11:19:26.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.817+0000 7f0d01ffb640 1 -- 172.21.15.104:0/4115142224 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d0c143000 con 0x7f0d0c1d7010 2024-03-29T11:19:26.806 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d01ffb640 1 -- 172.21.15.104:0/4115142224 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1062337736 0 0) 0x7f0cf0004eb0 con 0x7f0d0c09ac40 2024-03-29T11:19:26.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d01ffb640 1 -- 172.21.15.104:0/4115142224 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d0c0acd00 con 0x7f0d0c09ac40 2024-03-29T11:19:26.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d01ffb640 1 -- 172.21.15.104:0/4115142224 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f0cfc0018f0 con 0x7f0d0c09ac40 2024-03-29T11:19:26.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d01ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:26.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d01ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:26.807 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d01ffb640 10 monclient(hunting): dump: 2024-03-29T11:19:26.807 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-03-29T11:19:26.807 INFO:teuthology.orchestra.run.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:26.807 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:26.807 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:26.807 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:26.807 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-03-29T11:19:26.807 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:26.807 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:26.807 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:26.808 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-03-29T11:19:26.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d01ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:26.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d01ffb640 1 -- 172.21.15.104:0/4115142224 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 603808404 0 0) 0x7f0d0c143000 con 0x7f0d0c1d7010 2024-03-29T11:19:26.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d01ffb640 1 -- 172.21.15.104:0/4115142224 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0cf0004eb0 con 0x7f0d0c1d7010 2024-03-29T11:19:26.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d01ffb640 1 -- 172.21.15.104:0/4115142224 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2343607006 0 0) 0x7f0d0c1397e0 con 0x7f0d0c142380 2024-03-29T11:19:26.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d01ffb640 1 -- 172.21.15.104:0/4115142224 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d0c143000 con 0x7f0d0c142380 2024-03-29T11:19:26.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d01ffb640 1 -- 172.21.15.104:0/4115142224 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f0d04001e70 con 0x7f0d0c1d7010 2024-03-29T11:19:26.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d01ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:26.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d01ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:26.808 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d01ffb640 10 monclient(hunting): dump: 2024-03-29T11:19:26.808 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-03-29T11:19:26.808 INFO:teuthology.orchestra.run.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:26.808 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:26.808 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:26.808 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:26.809 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-03-29T11:19:26.809 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:26.809 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:26.809 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:26.809 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-03-29T11:19:26.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d01ffb640 1 -- 172.21.15.104:0/4115142224 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f0d08001290 con 0x7f0d0c142380 2024-03-29T11:19:26.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d01ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:26.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d01ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:26.809 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d01ffb640 10 monclient(hunting): dump: 2024-03-29T11:19:26.809 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-03-29T11:19:26.809 INFO:teuthology.orchestra.run.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:26.809 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:26.809 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:26.809 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:26.809 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-03-29T11:19:26.810 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:26.810 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:26.810 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:26.810 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-03-29T11:19:26.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d01ffb640 1 -- 172.21.15.104:0/4115142224 <== mon.2 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3759624272 0 0) 0x7f0cf0004eb0 con 0x7f0d0c1d7010 2024-03-29T11:19:26.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d01ffb640 1 -- 172.21.15.104:0/4115142224 >> v1:172.21.15.119:6789/0 conn(0x7f0d0c09ac40 legacy=0x7f0d0c1d7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:26.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d01ffb640 1 -- 172.21.15.104:0/4115142224 >> v1:172.21.15.104:6789/0 conn(0x7f0d0c142380 legacy=0x7f0d0c1389c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:26.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d01ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:19:26.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d01ffb640 1 monclient: found mon.c 2024-03-29T11:19:26.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d01ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:26.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d01ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:19:26.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d01ffb640 1 -- 172.21.15.104:0/4115142224 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d0c0ace90 con 0x7f0d0c1d7010 2024-03-29T11:19:26.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d01ffb640 10 monclient: _finish_auth 0 2024-03-29T11:19:26.810 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d01ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:19:26.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d01ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:26.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d1326e640 5 monclient: authenticate success, global_id 4187 2024-03-29T11:19:26.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d1326e640 10 monclient: _renew_subs 2024-03-29T11:19:26.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d1326e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:19:26.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d1326e640 1 -- 172.21.15.104:0/4115142224 --> v1:172.21.15.104:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d0c0ad080 con 0x7f0d0c1d7010 2024-03-29T11:19:26.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d01ffb640 1 -- 172.21.15.104:0/4115142224 <== mon.2 v1:172.21.15.104:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d04001fd0 con 0x7f0d0c1d7010 2024-03-29T11:19:26.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d01ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:26.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d01ffb640 1 -- 172.21.15.104:0/4115142224 <== mon.2 v1:172.21.15.104:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f0d04003dc0 con 0x7f0d0c1d7010 2024-03-29T11:19:26.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d01ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:26.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d01ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:26.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d037fe640 4 set_mon_vals no callback set 2024-03-29T11:19:26.811 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d01ffb640 10 monclient: dump: 2024-03-29T11:19:26.811 INFO:teuthology.orchestra.run.smithi104.stderr:epoch 1 2024-03-29T11:19:26.811 INFO:teuthology.orchestra.run.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:26.811 INFO:teuthology.orchestra.run.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:26.811 INFO:teuthology.orchestra.run.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:26.812 INFO:teuthology.orchestra.run.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:26.812 INFO:teuthology.orchestra.run.smithi104.stderr:election_strategy: 1 2024-03-29T11:19:26.812 INFO:teuthology.orchestra.run.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:26.812 INFO:teuthology.orchestra.run.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:26.812 INFO:teuthology.orchestra.run.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:26.812 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-03-29T11:19:26.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d1326e640 10 client.4187.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:26.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d1326e640 10 monclient: _renew_subs 2024-03-29T11:19:26.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d01ffb640 1 -- 172.21.15.104:0/4115142224 <== mon.2 v1:172.21.15.104:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f0d040042b0 con 0x7f0d0c1d7010 2024-03-29T11:19:26.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d1326e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:19:26.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d1326e640 1 -- 172.21.15.104:0/4115142224 --> v1:172.21.15.104:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d0c140180 con 0x7f0d0c1d7010 2024-03-29T11:19:26.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d1326e640 1 librados: init done 2024-03-29T11:19:26.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d01ffb640 1 -- 172.21.15.104:0/4115142224 <== mon.2 v1:172.21.15.104:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f0d04004eb0 con 0x7f0d0c1d7010 2024-03-29T11:19:26.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d01ffb640 10 client.4187.objecter ms_dispatch 0x7f0d0c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:26.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d01ffb640 3 client.4187.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:26.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d01ffb640 3 client.4187.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:26.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.818+0000 7f0d01ffb640 20 client.4187.objecter dump_active .. 0 homeless 2024-03-29T11:19:26.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.819+0000 7f0d1326e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:26.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.819+0000 7f0d1326e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:26.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.819+0000 7f0d1326e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:19:26.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.819+0000 7f0d1326e640 1 -- 172.21.15.104:0/4115142224 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d0c140430 con 0x7f0d0c1d7010 2024-03-29T11:19:26.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.821+0000 7f0d01ffb640 1 -- 172.21.15.104:0/4115142224 <== mon.2 v1:172.21.15.104:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f0d04005070 con 0x7f0d0c1d7010 2024-03-29T11:19:26.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.821+0000 7f0d01ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:26.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.821+0000 7f0d01ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:26.823 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-29T11:19:26.846+0000 7f0ede043040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:19:26.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.881+0000 7f0d1326e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:26.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.881+0000 7f0d1326e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:26.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.881+0000 7f0d1326e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:19:26.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.881+0000 7f0d1326e640 1 -- 172.21.15.104:0/4115142224 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d0c142dd0 con 0x7f0d0c1d7010 2024-03-29T11:19:26.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.882+0000 7f0d01ffb640 1 -- 172.21.15.104:0/4115142224 <== mon.2 v1:172.21.15.104:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f0d040048f0 con 0x7f0d0c1d7010 2024-03-29T11:19:26.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.882+0000 7f0d01ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:26.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.882+0000 7f0d01ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:26.859 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-03-29T11:19:26.859 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:26.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.882+0000 7f0d1326e640 10 librados: watch_flush enter 2024-03-29T11:19:26.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.882+0000 7f0d1326e640 10 librados: watch_flush exit 2024-03-29T11:19:26.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.882+0000 7f0d1326e640 20 client.4187.objecter shutdown clearing up homeless session... 2024-03-29T11:19:26.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.882+0000 7f0d1326e640 10 client.4187.objecter successfully canceled tick 2024-03-29T11:19:26.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.883+0000 7f0d1326e640 10 monclient: shutdown 2024-03-29T11:19:26.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.883+0000 7f0d1326e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:26.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.883+0000 7f0d1326e640 1 -- 172.21.15.104:0/4115142224 >> v1:172.21.15.104:6790/0 conn(0x7f0d0c1d7010 legacy=0x7f0d0c1390d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:26.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.883+0000 7f0d1326e640 1 -- 172.21.15.104:0/4115142224 shutdown_connections 2024-03-29T11:19:26.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.883+0000 7f0d1326e640 1 -- 172.21.15.104:0/4115142224 >> 172.21.15.104:0/4115142224 conn(0x7f0d0c066b00 msgr2=0x7f0d0c0b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:26.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.883+0000 7f0d1326e640 1 -- 172.21.15.104:0/4115142224 shutdown_connections 2024-03-29T11:19:26.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.883+0000 7f0d1326e640 1 -- 172.21.15.104:0/4115142224 wait complete. 2024-03-29T11:19:26.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-03-29T11:19:26.883+0000 7f0d1326e640 1 librados: shutdown 2024-03-29T11:19:26.870 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T11:19:26.870 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T11:19:26.889 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-03-29T11:19:26.912+0000 7f32f3443040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:19:26.921 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-29T11:19:26.945+0000 7f0ede043040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:19:26.982 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-03-29T11:19:27.005+0000 7f32f3443040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:19:27.076 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-29T11:19:27.099+0000 7f0ede043040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:19:27.117 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-29T11:19:27.141+0000 7f0ede043040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:19:27.139 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-03-29T11:19:27.162+0000 7f32f3443040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:19:27.180 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-03-29T11:19:27.203+0000 7f32f3443040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:19:27.242 INFO:tasks.ceph.osd.1.smithi104.stderr:2024-03-29T11:19:27.265+0000 7fea36c9b640 -1 Falling back to public interface 2024-03-29T11:19:27.249 INFO:tasks.ceph.osd.3.smithi104.stderr:2024-03-29T11:19:27.272+0000 7f16b5610640 -1 Falling back to public interface 2024-03-29T11:19:27.250 INFO:tasks.ceph.osd.2.smithi104.stderr:2024-03-29T11:19:27.273+0000 7f87cfe48640 -1 Falling back to public interface 2024-03-29T11:19:27.254 INFO:tasks.ceph.osd.0.smithi104.stderr:2024-03-29T11:19:27.277+0000 7f65e582a640 -1 Falling back to public interface 2024-03-29T11:19:27.271 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-29T11:19:27.295+0000 7f0ede043040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:19:27.275 INFO:tasks.ceph.osd.7.smithi119.stderr:2024-03-29T11:19:27.298+0000 7fc50380e640 -1 Falling back to public interface 2024-03-29T11:19:27.280 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-03-29T11:19:27.303+0000 7ffabb80d640 -1 Falling back to public interface 2024-03-29T11:19:27.280 INFO:tasks.ceph.osd.4.smithi119.stderr:2024-03-29T11:19:27.303+0000 7fdb3a013640 -1 Falling back to public interface 2024-03-29T11:19:27.281 INFO:tasks.ceph.osd.5.smithi119.stderr:2024-03-29T11:19:27.303+0000 7f505e815640 -1 Falling back to public interface 2024-03-29T11:19:27.335 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-03-29T11:19:27.358+0000 7f32f3443040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:19:27.674 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:27.764 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.787+0000 7fcedbf24640 10 monclient: get_monmap_and_config 2024-03-29T11:19:27.764 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.787+0000 7fcedbf24640 10 monclient: build_initial_monmap 2024-03-29T11:19:27.764 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.787+0000 7fcedbf24640 10 monclient: monmap: 2024-03-29T11:19:27.764 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:19:27.764 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:27.764 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:19:27.787954+0000 2024-03-29T11:19:27.764 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:19:27.787954+0000 2024-03-29T11:19:27.764 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:27.764 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:19:27.764 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:19:27.765 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:19:27.765 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:19:27.765 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:19:27.765 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.787+0000 7fcedbf24640 1 Processor -- start 2024-03-29T11:19:27.765 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.787+0000 7fcedbf24640 1 -- start start 2024-03-29T11:19:27.765 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.787+0000 7fcedbf24640 10 monclient: init 2024-03-29T11:19:27.773 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.788+0000 7fcedbf24640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:27.773 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.788+0000 7fcedbf24640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:27.773 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.788+0000 7fcedbf24640 10 monclient(hunting): picked mon.noname-c con 0x7fced41457f0 addr v1:172.21.15.119:6789/0 2024-03-29T11:19:27.773 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.788+0000 7fcedbf24640 10 monclient(hunting): picked mon.noname-b con 0x7fced413f650 addr v1:172.21.15.104:6790/0 2024-03-29T11:19:27.773 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.788+0000 7fcedbf24640 10 monclient(hunting): picked mon.noname-a con 0x7fced40a9d70 addr v1:172.21.15.104:6789/0 2024-03-29T11:19:27.773 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.788+0000 7fcedbf24640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fced40aaef0 con 0x7fced40a9d70 2024-03-29T11:19:27.774 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.788+0000 7fcedbf24640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fced4145bd0 con 0x7fced41457f0 2024-03-29T11:19:27.774 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.788+0000 7fcedbf24640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fced4145eb0 con 0x7fced413f650 2024-03-29T11:19:27.774 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.788+0000 7fcedbf24640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:27.774 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.788+0000 7fcedbf24640 10 monclient(hunting): authenticate will time out at 949.039501s 2024-03-29T11:19:27.774 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.788+0000 7fceda4c1640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fced40a9d70 0x7fced414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:42420/0 (socket says 172.21.15.104:42420) 2024-03-29T11:19:27.774 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.788+0000 7fceda4c1640 1 -- 172.21.15.104:0/3563242485 learned_addr learned my addr 172.21.15.104:0/3563242485 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:19:27.774 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.788+0000 7fced94bf640 1 --1- 172.21.15.104:0/3563242485 >> v1:172.21.15.104:6790/0 conn(0x7fced413f650 0x7fced40b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:40082/0 (socket says 172.21.15.104:40082) 2024-03-29T11:19:27.774 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.788+0000 7fced8cbe640 1 -- 172.21.15.104:0/3563242485 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2942523341 0 0) 0x7fced40aaef0 con 0x7fced40a9d70 2024-03-29T11:19:27.774 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.788+0000 7fced8cbe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:27.774 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.788+0000 7fced8cbe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:27.774 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.788+0000 7fced8cbe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:27.774 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.788+0000 7fced8cbe640 10 monclient(hunting): my global_id is 4194 2024-03-29T11:19:27.774 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.788+0000 7fced8cbe640 1 -- 172.21.15.104:0/3563242485 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcec0003520 con 0x7fced40a9d70 2024-03-29T11:19:27.774 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.788+0000 7fced8cbe640 1 -- 172.21.15.104:0/3563242485 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1727546354 0 0) 0x7fced4145eb0 con 0x7fced413f650 2024-03-29T11:19:27.775 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.788+0000 7fced8cbe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:27.775 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.788+0000 7fced8cbe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:27.775 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.788+0000 7fced8cbe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:27.775 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.788+0000 7fced8cbe640 10 monclient(hunting): my global_id is 4190 2024-03-29T11:19:27.775 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.788+0000 7fced8cbe640 1 -- 172.21.15.104:0/3563242485 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fced40aaef0 con 0x7fced413f650 2024-03-29T11:19:27.775 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.788+0000 7fced8cbe640 1 -- 172.21.15.104:0/3563242485 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4226984555 0 0) 0x7fced4145bd0 con 0x7fced41457f0 2024-03-29T11:19:27.775 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.789+0000 7fced8cbe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:27.775 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.789+0000 7fced8cbe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:27.775 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.789+0000 7fced8cbe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:27.775 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.789+0000 7fced8cbe640 10 monclient(hunting): my global_id is 4180 2024-03-29T11:19:27.775 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.789+0000 7fced8cbe640 1 -- 172.21.15.104:0/3563242485 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fced4145eb0 con 0x7fced41457f0 2024-03-29T11:19:27.775 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.789+0000 7fced8cbe640 1 -- 172.21.15.104:0/3563242485 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1157586739 0 0) 0x7fcec0003520 con 0x7fced40a9d70 2024-03-29T11:19:27.775 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.789+0000 7fced8cbe640 1 -- 172.21.15.104:0/3563242485 >> v1:172.21.15.104:6790/0 conn(0x7fced413f650 legacy=0x7fced40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:27.776 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.789+0000 7fced8cbe640 1 -- 172.21.15.104:0/3563242485 >> v1:172.21.15.119:6789/0 conn(0x7fced41457f0 legacy=0x7fced413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:27.776 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.789+0000 7fced8cbe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:27.776 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.789+0000 7fced8cbe640 1 monclient: found mon.noname-a 2024-03-29T11:19:27.776 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.789+0000 7fced8cbe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:27.776 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.789+0000 7fced8cbe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.104:6789/0 2024-03-29T11:19:27.776 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.789+0000 7fced8cbe640 1 -- 172.21.15.104:0/3563242485 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced4146340 con 0x7fced40a9d70 2024-03-29T11:19:27.776 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.789+0000 7fced8cbe640 10 monclient: _finish_auth 0 2024-03-29T11:19:27.776 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.789+0000 7fced8cbe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:27.776 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.789+0000 7fced8cbe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:27.776 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.789+0000 7fced8cbe640 1 -- 172.21.15.104:0/3563242485 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fcecc001a10 con 0x7fced40a9d70 2024-03-29T11:19:27.776 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.789+0000 7fcedbf24640 5 monclient: authenticate success, global_id 4194 2024-03-29T11:19:27.776 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.789+0000 7fcedbf24640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:27.776 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.789+0000 7fced8cbe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:27.776 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.789+0000 7fced8cbe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:27.776 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.789+0000 7fced8cbe640 10 monclient: dump: 2024-03-29T11:19:27.777 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:19:27.777 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:27.777 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:27.777 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:27.777 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:27.777 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:19:27.777 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:27.777 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:27.777 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:27.777 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:19:27.777 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.789+0000 7fcedbf24640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:27.777 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.789+0000 7fced8cbe640 1 -- 172.21.15.104:0/3563242485 <== mon.0 v1:172.21.15.104:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcecc001b70 con 0x7fced40a9d70 2024-03-29T11:19:27.777 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.789+0000 7fced8cbe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:27.777 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.789+0000 7fcedbf24640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:27.778 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.789+0000 7fced8cbe640 1 -- 172.21.15.104:0/3563242485 <== mon.0 v1:172.21.15.104:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fcecc001fc0 con 0x7fced40a9d70 2024-03-29T11:19:27.778 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.789+0000 7fced8cbe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:27.778 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.789+0000 7fcedbf24640 4 set_mon_vals no callback set 2024-03-29T11:19:27.778 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.789+0000 7fced8cbe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:27.778 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.789+0000 7fced8cbe640 10 monclient: dump: 2024-03-29T11:19:27.778 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:19:27.778 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:27.778 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:27.778 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:27.778 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:27.778 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:19:27.778 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:27.778 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:27.778 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:27.779 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:19:27.779 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.789+0000 7fcedbf24640 10 monclient: shutdown 2024-03-29T11:19:27.779 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.789+0000 7fcedbf24640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:27.779 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.789+0000 7fcedbf24640 1 -- 172.21.15.104:0/3563242485 >> v1:172.21.15.104:6789/0 conn(0x7fced40a9d70 legacy=0x7fced414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:27.779 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.789+0000 7fcedbf24640 1 -- 172.21.15.104:0/3563242485 shutdown_connections 2024-03-29T11:19:27.779 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.789+0000 7fcedbf24640 1 -- 172.21.15.104:0/3563242485 >> 172.21.15.104:0/3563242485 conn(0x7fced4066a00 msgr2=0x7fced4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:27.779 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.789+0000 7fcedbf24640 1 -- 172.21.15.104:0/3563242485 shutdown_connections 2024-03-29T11:19:27.779 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.789+0000 7fcedbf24640 1 -- 172.21.15.104:0/3563242485 wait complete. 2024-03-29T11:19:27.779 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.790+0000 7fcedbf24640 10 monclient: build_initial_monmap 2024-03-29T11:19:27.779 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.790+0000 7fcedbf24640 10 monclient: monmap: 2024-03-29T11:19:27.779 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:19:27.779 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:27.780 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:19:27.780 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:19:27.780 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:27.780 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:19:27.780 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:19:27.780 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:19:27.780 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:19:27.780 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:19:27.780 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.790+0000 7fcedbf24640 1 librados: starting msgr at 2024-03-29T11:19:27.780 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.790+0000 7fcedbf24640 1 librados: starting objecter 2024-03-29T11:19:27.780 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.790+0000 7fcedbf24640 1 Processor -- start 2024-03-29T11:19:27.780 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.790+0000 7fcedbf24640 1 -- start start 2024-03-29T11:19:27.780 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.790+0000 7fcedbf24640 1 librados: setting wanted keys 2024-03-29T11:19:27.780 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.790+0000 7fcedbf24640 1 librados: calling monclient init 2024-03-29T11:19:27.781 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.790+0000 7fcedbf24640 10 monclient: init 2024-03-29T11:19:27.781 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.791+0000 7fcedbf24640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:27.781 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.791+0000 7fcedbf24640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:27.781 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.791+0000 7fcedbf24640 10 monclient(hunting): picked mon.noname-b con 0x7fced40a9d70 addr v1:172.21.15.119:6789/0 2024-03-29T11:19:27.781 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.791+0000 7fcedbf24640 10 monclient(hunting): picked mon.noname-c con 0x7fced413f650 addr v1:172.21.15.104:6790/0 2024-03-29T11:19:27.781 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.791+0000 7fcedbf24640 10 monclient(hunting): picked mon.noname-a con 0x7fced41457f0 addr v1:172.21.15.104:6789/0 2024-03-29T11:19:27.781 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.791+0000 7fcedbf24640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fced4141da0 con 0x7fced41457f0 2024-03-29T11:19:27.781 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.791+0000 7fcedbf24640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fced4142080 con 0x7fced40a9d70 2024-03-29T11:19:27.781 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.791+0000 7fcedbf24640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fced413fda0 con 0x7fced413f650 2024-03-29T11:19:27.781 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.791+0000 7fcedbf24640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:27.781 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.791+0000 7fcedbf24640 10 monclient(hunting): authenticate will time out at 949.042477s 2024-03-29T11:19:27.781 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.791+0000 7fced94bf640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7fced413f650 0x7fced41402c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:40086/0 (socket says 172.21.15.104:40086) 2024-03-29T11:19:27.781 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.791+0000 7fced94bf640 1 -- 172.21.15.104:0/3539522489 learned_addr learned my addr 172.21.15.104:0/3539522489 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:19:27.781 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.791+0000 7fceca7fc640 10 client.?.objecter ms_handle_connect 0x7fced41457f0 2024-03-29T11:19:27.781 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.791+0000 7fceca7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:27.782 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.791+0000 7fceca7fc640 10 client.?.objecter ms_handle_connect 0x7fced413f650 2024-03-29T11:19:27.782 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.791+0000 7fceca7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:27.782 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.791+0000 7fceca7fc640 10 client.?.objecter ms_handle_connect 0x7fced40a9d70 2024-03-29T11:19:27.782 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.791+0000 7fceca7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:27.782 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.791+0000 7fceca7fc640 1 -- 172.21.15.104:0/3539522489 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3188104917 0 0) 0x7fced413fda0 con 0x7fced413f650 2024-03-29T11:19:27.782 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.791+0000 7fceca7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:27.782 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.791+0000 7fceca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:27.782 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.791+0000 7fceca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:27.782 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.791+0000 7fceca7fc640 10 monclient(hunting): my global_id is 4193 2024-03-29T11:19:27.782 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.791+0000 7fceca7fc640 1 -- 172.21.15.104:0/3539522489 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fceb8004eb0 con 0x7fced413f650 2024-03-29T11:19:27.782 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.791+0000 7fceca7fc640 1 -- 172.21.15.104:0/3539522489 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3648502956 0 0) 0x7fced4141da0 con 0x7fced41457f0 2024-03-29T11:19:27.782 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.791+0000 7fceca7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:27.782 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.791+0000 7fceca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:27.782 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.791+0000 7fceca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:27.782 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.791+0000 7fceca7fc640 10 monclient(hunting): my global_id is 4197 2024-03-29T11:19:27.783 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.791+0000 7fceca7fc640 1 -- 172.21.15.104:0/3539522489 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fced413fda0 con 0x7fced41457f0 2024-03-29T11:19:27.783 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.791+0000 7fceca7fc640 1 -- 172.21.15.104:0/3539522489 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4213918562 0 0) 0x7fced4142080 con 0x7fced40a9d70 2024-03-29T11:19:27.783 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.791+0000 7fceca7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:27.783 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.791+0000 7fceca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:27.783 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.791+0000 7fceca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:27.783 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.791+0000 7fceca7fc640 10 monclient(hunting): my global_id is 4183 2024-03-29T11:19:27.783 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.791+0000 7fceca7fc640 1 -- 172.21.15.104:0/3539522489 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fced4141da0 con 0x7fced40a9d70 2024-03-29T11:19:27.783 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.791+0000 7fceca7fc640 1 -- 172.21.15.104:0/3539522489 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 784499973 0 0) 0x7fceb8004eb0 con 0x7fced413f650 2024-03-29T11:19:27.783 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.791+0000 7fceca7fc640 1 -- 172.21.15.104:0/3539522489 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fced4142080 con 0x7fced413f650 2024-03-29T11:19:27.783 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.791+0000 7fceca7fc640 1 -- 172.21.15.104:0/3539522489 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1787484466 0 0) 0x7fced4141da0 con 0x7fced40a9d70 2024-03-29T11:19:27.783 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.791+0000 7fceca7fc640 1 -- 172.21.15.104:0/3539522489 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fceb8004eb0 con 0x7fced40a9d70 2024-03-29T11:19:27.783 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.791+0000 7fceca7fc640 1 -- 172.21.15.104:0/3539522489 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fced00012d0 con 0x7fced413f650 2024-03-29T11:19:27.783 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.791+0000 7fceca7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:27.783 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.791+0000 7fceca7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:27.783 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.791+0000 7fceca7fc640 10 monclient(hunting): dump: 2024-03-29T11:19:27.783 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:19:27.783 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:27.784 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:27.784 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:27.784 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:27.784 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:19:27.784 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:27.784 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:27.784 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:27.784 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:19:27.784 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.792+0000 7fceca7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:27.784 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.792+0000 7fceca7fc640 1 -- 172.21.15.104:0/3539522489 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1450910679 0 0) 0x7fced413fda0 con 0x7fced41457f0 2024-03-29T11:19:27.784 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.792+0000 7fceca7fc640 1 -- 172.21.15.104:0/3539522489 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fced4141da0 con 0x7fced41457f0 2024-03-29T11:19:27.784 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.792+0000 7fceca7fc640 1 -- 172.21.15.104:0/3539522489 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fcec4001790 con 0x7fced40a9d70 2024-03-29T11:19:27.784 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.792+0000 7fceca7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:27.784 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.792+0000 7fceca7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:27.785 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.792+0000 7fceca7fc640 10 monclient(hunting): dump: 2024-03-29T11:19:27.785 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:19:27.785 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:27.785 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:27.785 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:27.785 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:27.785 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:19:27.785 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:27.785 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:27.785 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:27.785 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:19:27.785 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.792+0000 7fceca7fc640 1 -- 172.21.15.104:0/3539522489 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fcecc001830 con 0x7fced41457f0 2024-03-29T11:19:27.785 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.792+0000 7fceca7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:27.785 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.792+0000 7fceca7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:27.785 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.792+0000 7fceca7fc640 10 monclient(hunting): dump: 2024-03-29T11:19:27.785 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:19:27.785 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:27.786 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:27.786 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:27.786 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:27.786 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:19:27.786 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:27.786 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:27.786 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:27.786 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:19:27.786 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.792+0000 7fceca7fc640 1 -- 172.21.15.104:0/3539522489 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3904528523 0 0) 0x7fceb8004eb0 con 0x7fced40a9d70 2024-03-29T11:19:27.786 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.792+0000 7fceca7fc640 1 -- 172.21.15.104:0/3539522489 >> v1:172.21.15.104:6790/0 conn(0x7fced413f650 legacy=0x7fced41402c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:27.786 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.792+0000 7fceca7fc640 1 -- 172.21.15.104:0/3539522489 >> v1:172.21.15.104:6789/0 conn(0x7fced41457f0 legacy=0x7fced41e1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:27.786 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.792+0000 7fceca7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:19:27.786 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.792+0000 7fceca7fc640 1 monclient: found mon.b 2024-03-29T11:19:27.786 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.792+0000 7fceca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:27.787 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.792+0000 7fceca7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:19:27.787 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.792+0000 7fceca7fc640 1 -- 172.21.15.104:0/3539522489 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced4138d30 con 0x7fced40a9d70 2024-03-29T11:19:27.787 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.792+0000 7fceca7fc640 10 monclient: _finish_auth 0 2024-03-29T11:19:27.787 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.792+0000 7fceca7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:19:27.787 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.792+0000 7fceca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:27.787 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.792+0000 7fcedbf24640 5 monclient: authenticate success, global_id 4183 2024-03-29T11:19:27.787 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.792+0000 7fcedbf24640 10 monclient: _renew_subs 2024-03-29T11:19:27.787 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.792+0000 7fcedbf24640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:19:27.787 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.792+0000 7fcedbf24640 1 -- 172.21.15.104:0/3539522489 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fced413b680 con 0x7fced40a9d70 2024-03-29T11:19:27.787 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.792+0000 7fcedbf24640 10 client.4183.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:27.787 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.792+0000 7fcedbf24640 10 monclient: _renew_subs 2024-03-29T11:19:27.787 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.792+0000 7fcedbf24640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:19:27.787 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.792+0000 7fcedbf24640 1 -- 172.21.15.104:0/3539522489 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fced4140bb0 con 0x7fced40a9d70 2024-03-29T11:19:27.787 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.792+0000 7fcedbf24640 1 librados: init done 2024-03-29T11:19:27.787 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.792+0000 7fceca7fc640 1 -- 172.21.15.104:0/3539522489 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcec4001d60 con 0x7fced40a9d70 2024-03-29T11:19:27.787 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.792+0000 7fceca7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:27.788 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.792+0000 7fceca7fc640 1 -- 172.21.15.104:0/3539522489 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fcec40037b0 con 0x7fced40a9d70 2024-03-29T11:19:27.788 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.792+0000 7fceca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:27.788 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.792+0000 7fceca7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:27.788 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.792+0000 7fcecbfff640 4 set_mon_vals no callback set 2024-03-29T11:19:27.788 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.792+0000 7fceca7fc640 10 monclient: dump: 2024-03-29T11:19:27.788 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:19:27.788 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:27.788 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:27.788 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:27.788 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:27.788 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:19:27.788 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:27.788 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:27.788 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:27.788 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:19:27.788 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.792+0000 7fceca7fc640 1 -- 172.21.15.104:0/3539522489 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fcec4003ce0 con 0x7fced40a9d70 2024-03-29T11:19:27.789 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.792+0000 7fceca7fc640 1 -- 172.21.15.104:0/3539522489 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7fcec4005640 con 0x7fced40a9d70 2024-03-29T11:19:27.789 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.792+0000 7fceca7fc640 10 client.4183.objecter ms_dispatch 0x7fced4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:27.789 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.792+0000 7fceca7fc640 3 client.4183.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:27.789 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.792+0000 7fceca7fc640 3 client.4183.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:27.789 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.792+0000 7fceca7fc640 20 client.4183.objecter dump_active .. 0 homeless 2024-03-29T11:19:27.789 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.793+0000 7fcedbf24640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:27.789 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.793+0000 7fcedbf24640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:27.789 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.793+0000 7fcedbf24640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:19:27.789 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.793+0000 7fcedbf24640 1 -- 172.21.15.104:0/3539522489 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcebc005080 con 0x7fced40a9d70 2024-03-29T11:19:27.789 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.795+0000 7fceca7fc640 1 -- 172.21.15.104:0/3539522489 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fcec4004fd0 con 0x7fced40a9d70 2024-03-29T11:19:27.789 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.795+0000 7fceca7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:27.789 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.795+0000 7fceca7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:27.833 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.856+0000 7fcedbf24640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:27.833 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.856+0000 7fcedbf24640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:27.833 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.856+0000 7fcedbf24640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:19:27.833 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.856+0000 7fcedbf24640 1 -- 172.21.15.104:0/3539522489 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcebc005330 con 0x7fced40a9d70 2024-03-29T11:19:27.834 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.857+0000 7fceca7fc640 1 -- 172.21.15.104:0/3539522489 <== mon.1 v1:172.21.15.119:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7fcec40052e0 con 0x7fced40a9d70 2024-03-29T11:19:27.834 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.857+0000 7fceca7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:27.834 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.857+0000 7fceca7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:27.834 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:19:27.834 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:27.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.858+0000 7fcedbf24640 10 librados: watch_flush enter 2024-03-29T11:19:27.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.858+0000 7fcedbf24640 10 librados: watch_flush exit 2024-03-29T11:19:27.836 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.858+0000 7fcedbf24640 20 client.4183.objecter shutdown clearing up homeless session... 2024-03-29T11:19:27.836 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.858+0000 7fcedbf24640 10 client.4183.objecter successfully canceled tick 2024-03-29T11:19:27.836 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.858+0000 7fcedbf24640 10 monclient: shutdown 2024-03-29T11:19:27.836 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.858+0000 7fcedbf24640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:27.836 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.858+0000 7fcedbf24640 1 -- 172.21.15.104:0/3539522489 >> v1:172.21.15.119:6789/0 conn(0x7fced40a9d70 legacy=0x7fced4142810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:27.836 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.858+0000 7fcedbf24640 1 -- 172.21.15.104:0/3539522489 shutdown_connections 2024-03-29T11:19:27.836 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.858+0000 7fcedbf24640 1 -- 172.21.15.104:0/3539522489 >> 172.21.15.104:0/3539522489 conn(0x7fced4066a00 msgr2=0x7fced40b4800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:27.836 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.858+0000 7fcedbf24640 1 -- 172.21.15.104:0/3539522489 shutdown_connections 2024-03-29T11:19:27.836 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.858+0000 7fcedbf24640 1 -- 172.21.15.104:0/3539522489 wait complete. 2024-03-29T11:19:27.836 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:27.858+0000 7fcedbf24640 1 librados: shutdown 2024-03-29T11:19:27.846 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:28.081 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-29T11:19:28.104+0000 7f0ede043040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:19:28.157 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-03-29T11:19:28.180+0000 7f32f3443040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:19:28.353 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-29T11:19:28.376+0000 7f0ede043040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:19:28.406 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-29T11:19:28.429+0000 7f0ede043040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:19:28.429 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-03-29T11:19:28.452+0000 7f32f3443040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:19:28.480 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-03-29T11:19:28.503+0000 7f32f3443040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:19:28.520 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-29T11:19:28.544+0000 7f0ede043040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:19:28.600 INFO:tasks.ceph.mgr.x.smithi119.stderr:2024-03-29T11:19:28.623+0000 7f32f3443040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:19:29.533 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-29T11:19:29.556+0000 7f0e7edd5640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-29T11:19:29.533 INFO:tasks.ceph.mgr.y.smithi104.stderr:2024-03-29T11:19:29.557+0000 7f0e7edd5640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-29T11:19:30.715 INFO:tasks.ceph.osd.2.smithi104.stderr:2024-03-29T11:19:30.738+0000 7f87cfe48640 -1 osd.2 0 log_to_monitors true 2024-03-29T11:19:30.753 INFO:tasks.ceph.osd.1.smithi104.stderr:2024-03-29T11:19:30.776+0000 7fea36c9b640 -1 osd.1 0 log_to_monitors true 2024-03-29T11:19:30.773 INFO:tasks.ceph.osd.0.smithi104.stderr:2024-03-29T11:19:30.796+0000 7f65e582a640 -1 osd.0 0 log_to_monitors true 2024-03-29T11:19:30.773 INFO:tasks.ceph.osd.7.smithi119.stderr:2024-03-29T11:19:30.796+0000 7fc50380e640 -1 osd.7 0 log_to_monitors true 2024-03-29T11:19:30.798 INFO:tasks.ceph.osd.3.smithi104.stderr:2024-03-29T11:19:30.821+0000 7f16b5610640 -1 osd.3 0 log_to_monitors true 2024-03-29T11:19:30.807 INFO:tasks.ceph.osd.6.smithi119.stderr:2024-03-29T11:19:30.830+0000 7ffabb80d640 -1 osd.6 0 log_to_monitors true 2024-03-29T11:19:30.811 INFO:tasks.ceph.osd.5.smithi119.stderr:2024-03-29T11:19:30.834+0000 7f505e815640 -1 osd.5 0 log_to_monitors true 2024-03-29T11:19:30.820 INFO:tasks.ceph.osd.4.smithi119.stderr:2024-03-29T11:19:30.843+0000 7fdb3a013640 -1 osd.4 0 log_to_monitors true 2024-03-29T11:19:34.652 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:34.809 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.831+0000 7f000c08f640 10 monclient: get_monmap_and_config 2024-03-29T11:19:34.809 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.831+0000 7f000c08f640 10 monclient: build_initial_monmap 2024-03-29T11:19:34.809 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.831+0000 7f000c08f640 10 monclient: monmap: 2024-03-29T11:19:34.809 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:19:34.809 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:34.809 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:19:34.833313+0000 2024-03-29T11:19:34.809 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:19:34.833313+0000 2024-03-29T11:19:34.809 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:34.810 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:19:34.810 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:19:34.810 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:19:34.810 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:19:34.810 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:19:34.811 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.832+0000 7f000c08f640 1 Processor -- start 2024-03-29T11:19:34.811 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.832+0000 7f000c08f640 1 -- start start 2024-03-29T11:19:34.811 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.832+0000 7f000c08f640 10 monclient: init 2024-03-29T11:19:34.811 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.833+0000 7f000c08f640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:34.812 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.833+0000 7f000c08f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:19:34.812 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.833+0000 7f000c08f640 10 monclient(hunting): picked mon.noname-b con 0x7f00041457f0 addr v1:172.21.15.104:6790/0 2024-03-29T11:19:34.812 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.833+0000 7f000c08f640 10 monclient(hunting): picked mon.noname-a con 0x7f000413f650 addr v1:172.21.15.104:6789/0 2024-03-29T11:19:34.812 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.833+0000 7f000c08f640 10 monclient(hunting): picked mon.noname-c con 0x7f00040a9d70 addr v1:172.21.15.119:6789/0 2024-03-29T11:19:34.812 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.833+0000 7f000c08f640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00040aaef0 con 0x7f000413f650 2024-03-29T11:19:34.812 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.833+0000 7f000c08f640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0004145bd0 con 0x7f00040a9d70 2024-03-29T11:19:34.812 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.833+0000 7f000c08f640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0004145eb0 con 0x7f00041457f0 2024-03-29T11:19:34.812 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.833+0000 7f000c08f640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:34.812 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.833+0000 7f000c08f640 10 monclient(hunting): authenticate will time out at 956.085525s 2024-03-29T11:19:34.812 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.833+0000 7f0009e2b640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f00041457f0 0x7f000413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:41676/0 (socket says 172.21.15.104:41676) 2024-03-29T11:19:34.812 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.833+0000 7f0009e2b640 1 -- 172.21.15.104:0/3295307708 learned_addr learned my addr 172.21.15.104:0/3295307708 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:19:34.813 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.834+0000 7f0008e29640 1 -- 172.21.15.104:0/3295307708 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 797012283 0 0) 0x7f00040aaef0 con 0x7f000413f650 2024-03-29T11:19:34.813 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.834+0000 7f0008e29640 10 monclient(hunting): _negotiate 2024-03-29T11:19:34.813 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.834+0000 7f0008e29640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:34.813 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.834+0000 7f0008e29640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:34.813 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.834+0000 7f0008e29640 10 monclient(hunting): my global_id is 4242 2024-03-29T11:19:34.814 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.834+0000 7f0008e29640 1 -- 172.21.15.104:0/3295307708 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efff0003520 con 0x7f000413f650 2024-03-29T11:19:34.814 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.835+0000 7f0008e29640 1 -- 172.21.15.104:0/3295307708 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 568250865 0 0) 0x7f0004145eb0 con 0x7f00041457f0 2024-03-29T11:19:34.814 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.835+0000 7f0008e29640 10 monclient(hunting): _negotiate 2024-03-29T11:19:34.814 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.835+0000 7f0008e29640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:34.814 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.835+0000 7f0008e29640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:34.814 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.835+0000 7f0008e29640 10 monclient(hunting): my global_id is 4238 2024-03-29T11:19:34.814 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.835+0000 7f0008e29640 1 -- 172.21.15.104:0/3295307708 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00040aaef0 con 0x7f00041457f0 2024-03-29T11:19:34.814 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.835+0000 7f0008e29640 1 -- 172.21.15.104:0/3295307708 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2354634527 0 0) 0x7f0004145bd0 con 0x7f00040a9d70 2024-03-29T11:19:34.814 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.835+0000 7f0008e29640 10 monclient(hunting): _negotiate 2024-03-29T11:19:34.814 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.835+0000 7f0008e29640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:34.814 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.835+0000 7f0008e29640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:34.814 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.835+0000 7f0008e29640 10 monclient(hunting): my global_id is 4222 2024-03-29T11:19:34.814 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.835+0000 7f0008e29640 1 -- 172.21.15.104:0/3295307708 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0004145eb0 con 0x7f00040a9d70 2024-03-29T11:19:34.814 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.835+0000 7f0008e29640 1 -- 172.21.15.104:0/3295307708 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1670910300 0 0) 0x7f0004145eb0 con 0x7f00040a9d70 2024-03-29T11:19:34.815 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.835+0000 7f0008e29640 1 -- 172.21.15.104:0/3295307708 >> v1:172.21.15.104:6790/0 conn(0x7f00041457f0 legacy=0x7f000413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:34.815 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.835+0000 7f0008e29640 1 -- 172.21.15.104:0/3295307708 >> v1:172.21.15.104:6789/0 conn(0x7f000413f650 legacy=0x7f00040b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:34.815 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.835+0000 7f0008e29640 10 monclient: _finish_hunting 0 2024-03-29T11:19:34.815 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.835+0000 7f0008e29640 1 monclient: found mon.noname-c 2024-03-29T11:19:34.815 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.835+0000 7f0008e29640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:34.815 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.835+0000 7f0008e29640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.119:6789/0 2024-03-29T11:19:34.815 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.835+0000 7f0008e29640 1 -- 172.21.15.104:0/3295307708 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0004146340 con 0x7f00040a9d70 2024-03-29T11:19:34.815 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.835+0000 7f0008e29640 10 monclient: _finish_auth 0 2024-03-29T11:19:34.815 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.835+0000 7f0008e29640 10 monclient: _check_auth_tickets 2024-03-29T11:19:34.815 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.835+0000 7f0008e29640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:34.815 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.835+0000 7f0008e29640 1 -- 172.21.15.104:0/3295307708 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7efff4001530 con 0x7f00040a9d70 2024-03-29T11:19:34.815 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.835+0000 7f0008e29640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:34.815 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.835+0000 7f0008e29640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:34.815 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.835+0000 7f0008e29640 10 monclient: dump: 2024-03-29T11:19:34.815 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:19:34.816 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:34.816 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:34.816 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:34.816 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:34.816 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:19:34.816 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:34.816 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:34.816 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:34.816 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:19:34.816 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.836+0000 7f000c08f640 5 monclient: authenticate success, global_id 4222 2024-03-29T11:19:34.816 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.836+0000 7f000c08f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:34.818 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.836+0000 7f0008e29640 1 -- 172.21.15.104:0/3295307708 <== mon.1 v1:172.21.15.119:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efff4001690 con 0x7f00040a9d70 2024-03-29T11:19:34.818 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.836+0000 7f0008e29640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:34.818 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.836+0000 7f000c08f640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:34.818 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.836+0000 7f000c08f640 4 set_mon_vals no callback set 2024-03-29T11:19:34.818 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.836+0000 7f0008e29640 1 -- 172.21.15.104:0/3295307708 <== mon.1 v1:172.21.15.119:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7efff4001a60 con 0x7f00040a9d70 2024-03-29T11:19:34.818 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.836+0000 7f0008e29640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:34.818 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.836+0000 7f0008e29640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:34.818 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.836+0000 7f0008e29640 10 monclient: dump: 2024-03-29T11:19:34.819 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:19:34.819 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:34.819 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:34.819 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:34.819 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:34.819 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:19:34.819 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:34.819 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:34.819 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:34.819 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:19:34.820 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.837+0000 7f000c08f640 10 monclient: shutdown 2024-03-29T11:19:34.820 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.837+0000 7f000c08f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:34.820 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.837+0000 7f000c08f640 1 -- 172.21.15.104:0/3295307708 >> v1:172.21.15.119:6789/0 conn(0x7f00040a9d70 legacy=0x7f000414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:34.820 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.837+0000 7f000c08f640 1 -- 172.21.15.104:0/3295307708 shutdown_connections 2024-03-29T11:19:34.820 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.837+0000 7f000c08f640 1 -- 172.21.15.104:0/3295307708 >> 172.21.15.104:0/3295307708 conn(0x7f0004066a00 msgr2=0x7f0004066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:34.820 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.838+0000 7f000c08f640 1 -- 172.21.15.104:0/3295307708 shutdown_connections 2024-03-29T11:19:34.820 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.838+0000 7f000c08f640 1 -- 172.21.15.104:0/3295307708 wait complete. 2024-03-29T11:19:34.820 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.839+0000 7f000c08f640 10 monclient: build_initial_monmap 2024-03-29T11:19:34.820 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.839+0000 7f000c08f640 10 monclient: monmap: 2024-03-29T11:19:34.820 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:19:34.820 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:34.820 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:19:34.821 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:19:34.821 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:34.821 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:19:34.821 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:19:34.821 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:19:34.821 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:19:34.821 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:19:34.821 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.839+0000 7f000c08f640 1 librados: starting msgr at 2024-03-29T11:19:34.821 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.839+0000 7f000c08f640 1 librados: starting objecter 2024-03-29T11:19:34.821 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.839+0000 7f000c08f640 1 Processor -- start 2024-03-29T11:19:34.821 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.839+0000 7f000c08f640 1 -- start start 2024-03-29T11:19:34.821 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.839+0000 7f000c08f640 1 librados: setting wanted keys 2024-03-29T11:19:34.821 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.839+0000 7f000c08f640 1 librados: calling monclient init 2024-03-29T11:19:34.821 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.839+0000 7f000c08f640 10 monclient: init 2024-03-29T11:19:34.822 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.840+0000 7f000c08f640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:34.822 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.840+0000 7f000c08f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:34.822 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.840+0000 7f000c08f640 10 monclient(hunting): picked mon.noname-b con 0x7f00040a9d70 addr v1:172.21.15.119:6789/0 2024-03-29T11:19:34.822 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.840+0000 7f000c08f640 10 monclient(hunting): picked mon.noname-c con 0x7f000413f650 addr v1:172.21.15.104:6790/0 2024-03-29T11:19:34.822 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.840+0000 7f000c08f640 10 monclient(hunting): picked mon.noname-a con 0x7f00041457f0 addr v1:172.21.15.104:6789/0 2024-03-29T11:19:34.822 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.840+0000 7f000c08f640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00041d7f90 con 0x7f00041457f0 2024-03-29T11:19:34.822 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.840+0000 7f000c08f640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00040ad2a0 con 0x7f00040a9d70 2024-03-29T11:19:34.822 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.840+0000 7f000c08f640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00041d7080 con 0x7f000413f650 2024-03-29T11:19:34.822 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.840+0000 7f000c08f640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:34.822 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.840+0000 7f000c08f640 10 monclient(hunting): authenticate will time out at 956.092506s 2024-03-29T11:19:34.822 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.840+0000 7f000a62c640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f00041457f0 0x7f00041e29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:56060/0 (socket says 172.21.15.104:56060) 2024-03-29T11:19:34.822 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.840+0000 7f000a62c640 1 -- 172.21.15.104:0/1984072672 learned_addr learned my addr 172.21.15.104:0/1984072672 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:19:34.825 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.841+0000 7efffa7fc640 10 client.?.objecter ms_handle_connect 0x7f00041457f0 2024-03-29T11:19:34.825 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.841+0000 7efffa7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:34.825 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.841+0000 7efffa7fc640 10 client.?.objecter ms_handle_connect 0x7f00040a9d70 2024-03-29T11:19:34.825 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.841+0000 7efffa7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:34.825 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.841+0000 7efffa7fc640 1 -- 172.21.15.104:0/1984072672 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2200350952 0 0) 0x7f00041d7f90 con 0x7f00041457f0 2024-03-29T11:19:34.825 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.841+0000 7efffa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:34.825 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.841+0000 7efffa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:34.826 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.841+0000 7efffa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:34.826 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.841+0000 7efffa7fc640 10 monclient(hunting): my global_id is 4245 2024-03-29T11:19:34.826 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.841+0000 7efffa7fc640 1 -- 172.21.15.104:0/1984072672 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effe8004eb0 con 0x7f00041457f0 2024-03-29T11:19:34.826 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.841+0000 7efffa7fc640 1 -- 172.21.15.104:0/1984072672 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1097908498 0 0) 0x7f00040ad2a0 con 0x7f00040a9d70 2024-03-29T11:19:34.826 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.841+0000 7efffa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:34.826 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.841+0000 7efffa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:34.826 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.841+0000 7efffa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:34.826 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.841+0000 7efffa7fc640 10 monclient(hunting): my global_id is 4225 2024-03-29T11:19:34.826 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.841+0000 7efffa7fc640 1 -- 172.21.15.104:0/1984072672 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effe80051d0 con 0x7f00040a9d70 2024-03-29T11:19:34.826 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.842+0000 7efffa7fc640 1 -- 172.21.15.104:0/1984072672 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2882007869 0 0) 0x7effe80051d0 con 0x7f00040a9d70 2024-03-29T11:19:34.826 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.842+0000 7efffa7fc640 1 -- 172.21.15.104:0/1984072672 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7effe8005640 con 0x7f00040a9d70 2024-03-29T11:19:34.826 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.842+0000 7efffa7fc640 1 -- 172.21.15.104:0/1984072672 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7efffc001bd0 con 0x7f00040a9d70 2024-03-29T11:19:34.826 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.842+0000 7efffa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:34.826 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.842+0000 7efffa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:34.827 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.842+0000 7efffa7fc640 10 monclient(hunting): dump: 2024-03-29T11:19:34.827 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:19:34.827 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:34.827 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:34.827 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:34.827 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:34.827 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:19:34.827 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:34.827 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:34.827 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:34.827 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:19:34.827 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.842+0000 7efffa7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:34.827 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.842+0000 7efffa7fc640 1 -- 172.21.15.104:0/1984072672 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3379683305 0 0) 0x7effe8004eb0 con 0x7f00041457f0 2024-03-29T11:19:34.827 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.842+0000 7efffa7fc640 1 -- 172.21.15.104:0/1984072672 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7effe80051d0 con 0x7f00041457f0 2024-03-29T11:19:34.827 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.842+0000 7efffa7fc640 1 -- 172.21.15.104:0/1984072672 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7efff4001a60 con 0x7f00041457f0 2024-03-29T11:19:34.827 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.842+0000 7efffa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:34.828 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.842+0000 7efffa7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:34.828 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.842+0000 7efffa7fc640 10 monclient(hunting): dump: 2024-03-29T11:19:34.828 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:19:34.828 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:34.828 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:34.828 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:34.828 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:34.828 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:19:34.828 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:34.828 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:34.828 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:34.828 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:19:34.828 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.842+0000 7efffa7fc640 10 client.?.objecter ms_handle_connect 0x7f000413f650 2024-03-29T11:19:34.828 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.842+0000 7efffa7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:34.828 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.842+0000 7efffa7fc640 1 -- 172.21.15.104:0/1984072672 <== mon.0 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1561536157 0 0) 0x7effe80051d0 con 0x7f00041457f0 2024-03-29T11:19:34.828 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.842+0000 7efffa7fc640 1 -- 172.21.15.104:0/1984072672 >> v1:172.21.15.104:6790/0 conn(0x7f000413f650 legacy=0x7f0004137cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:34.828 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.842+0000 7efffa7fc640 1 -- 172.21.15.104:0/1984072672 >> v1:172.21.15.119:6789/0 conn(0x7f00040a9d70 legacy=0x7f00041d7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:34.829 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.842+0000 7efffa7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:19:34.829 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.842+0000 7efffa7fc640 1 monclient: found mon.a 2024-03-29T11:19:34.829 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.842+0000 7efffa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:34.829 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.842+0000 7efffa7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:19:34.829 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.842+0000 7efffa7fc640 1 -- 172.21.15.104:0/1984072672 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0004142780 con 0x7f00041457f0 2024-03-29T11:19:34.829 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.842+0000 7efffa7fc640 10 monclient: _finish_auth 0 2024-03-29T11:19:34.829 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.842+0000 7efffa7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:19:34.829 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.842+0000 7efffa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:34.829 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.842+0000 7f000c08f640 5 monclient: authenticate success, global_id 4245 2024-03-29T11:19:34.829 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.842+0000 7f000c08f640 10 monclient: _renew_subs 2024-03-29T11:19:34.829 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.842+0000 7f000c08f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:19:34.829 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.842+0000 7f000c08f640 1 -- 172.21.15.104:0/1984072672 --> v1:172.21.15.104:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0004142a30 con 0x7f00041457f0 2024-03-29T11:19:34.829 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.842+0000 7f000c08f640 10 client.4245.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:34.829 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.842+0000 7f000c08f640 10 monclient: _renew_subs 2024-03-29T11:19:34.829 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.842+0000 7f000c08f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:19:34.830 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.842+0000 7f000c08f640 1 -- 172.21.15.104:0/1984072672 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f000413ff80 con 0x7f00041457f0 2024-03-29T11:19:34.830 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.842+0000 7f000c08f640 1 librados: init done 2024-03-29T11:19:34.830 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.843+0000 7efffa7fc640 1 -- 172.21.15.104:0/1984072672 <== mon.0 v1:172.21.15.104:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efff40038a0 con 0x7f00041457f0 2024-03-29T11:19:34.830 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.843+0000 7efffa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:34.830 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.843+0000 7efffbfff640 4 set_mon_vals no callback set 2024-03-29T11:19:34.830 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.844+0000 7f000c08f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:34.830 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.844+0000 7efffa7fc640 1 -- 172.21.15.104:0/1984072672 <== mon.0 v1:172.21.15.104:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7efff4003cf0 con 0x7f00041457f0 2024-03-29T11:19:34.830 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.844+0000 7f000c08f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:34.830 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.844+0000 7f000c08f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:19:34.830 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.844+0000 7f000c08f640 1 -- 172.21.15.104:0/1984072672 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effec005080 con 0x7f00041457f0 2024-03-29T11:19:34.830 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.844+0000 7efffa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:34.830 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.844+0000 7efffa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:34.830 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.844+0000 7efffa7fc640 10 monclient: dump: 2024-03-29T11:19:34.830 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:19:34.831 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:34.831 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:34.831 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:34.831 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:34.831 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:19:34.831 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:34.831 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:34.831 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:34.831 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:19:34.831 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.846+0000 7efffa7fc640 1 -- 172.21.15.104:0/1984072672 <== mon.0 v1:172.21.15.104:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7efff4003ed0 con 0x7f00041457f0 2024-03-29T11:19:34.831 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.847+0000 7f0009e2b640 10 monclient: get_auth_request con 0x7effe806e2b0 auth_method 0 2024-03-29T11:19:34.831 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.847+0000 7efffa7fc640 1 -- 172.21.15.104:0/1984072672 <== mon.0 v1:172.21.15.104:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7efff4084940 con 0x7f00041457f0 2024-03-29T11:19:34.831 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.847+0000 7efffa7fc640 10 client.4245.objecter ms_dispatch 0x7f0004000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:34.831 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.847+0000 7efffa7fc640 3 client.4245.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:34.831 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.847+0000 7efffa7fc640 3 client.4245.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:34.832 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.847+0000 7efffa7fc640 20 client.4245.objecter dump_active .. 0 homeless 2024-03-29T11:19:34.832 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.847+0000 7efffa7fc640 10 client.4245.objecter ms_handle_connect 0x7effe806e2b0 2024-03-29T11:19:34.838 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.860+0000 7efffa7fc640 1 -- 172.21.15.104:0/1984072672 <== mon.0 v1:172.21.15.104:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efff4050360 con 0x7f00041457f0 2024-03-29T11:19:34.838 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.860+0000 7efffa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:34.838 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:34.860+0000 7efffa7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:35.197 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:35.219+0000 7f000c08f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:35.197 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:35.219+0000 7f000c08f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:35.197 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:35.219+0000 7f000c08f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:19:35.197 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:35.219+0000 7f000c08f640 1 -- 172.21.15.104:0/1984072672 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7effec005330 con 0x7f00041457f0 2024-03-29T11:19:35.199 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:35.221+0000 7efffa7fc640 1 -- 172.21.15.104:0/1984072672 <== mon.0 v1:172.21.15.104:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7efff4050360 con 0x7f00041457f0 2024-03-29T11:19:35.199 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:35.221+0000 7efffa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:35.199 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:35.221+0000 7efffa7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:35.200 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:19:35.200 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:35.202 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:35.224+0000 7f000c08f640 10 librados: watch_flush enter 2024-03-29T11:19:35.203 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:35.225+0000 7f000c08f640 10 librados: watch_flush exit 2024-03-29T11:19:35.203 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:35.225+0000 7f000c08f640 20 client.4245.objecter shutdown clearing up homeless session... 2024-03-29T11:19:35.203 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:35.225+0000 7f000c08f640 10 client.4245.objecter successfully canceled tick 2024-03-29T11:19:35.203 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:35.225+0000 7f000c08f640 1 -- 172.21.15.104:0/1984072672 >> v1:172.21.15.104:6816/3341126660 conn(0x7effe806e2b0 legacy=0x7effe8070770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:35.203 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:35.225+0000 7f000c08f640 10 monclient: shutdown 2024-03-29T11:19:35.203 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:35.225+0000 7f000c08f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:35.203 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:35.225+0000 7f000c08f640 1 -- 172.21.15.104:0/1984072672 >> v1:172.21.15.104:6789/0 conn(0x7f00041457f0 legacy=0x7f00041e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:35.203 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:35.225+0000 7f000c08f640 1 -- 172.21.15.104:0/1984072672 shutdown_connections 2024-03-29T11:19:35.203 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:35.225+0000 7f000c08f640 1 -- 172.21.15.104:0/1984072672 >> 172.21.15.104:0/1984072672 conn(0x7f0004066a00 msgr2=0x7f00040b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:35.203 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:35.225+0000 7f000c08f640 1 -- 172.21.15.104:0/1984072672 shutdown_connections 2024-03-29T11:19:35.204 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:35.226+0000 7f000c08f640 1 -- 172.21.15.104:0/1984072672 wait complete. 2024-03-29T11:19:35.204 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:35.226+0000 7f000c08f640 1 librados: shutdown 2024-03-29T11:19:35.219 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:42.026 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:42.172 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.195+0000 7fe44842a640 10 monclient: get_monmap_and_config 2024-03-29T11:19:42.172 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.195+0000 7fe44842a640 10 monclient: build_initial_monmap 2024-03-29T11:19:42.172 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.195+0000 7fe44842a640 10 monclient: monmap: 2024-03-29T11:19:42.172 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:19:42.172 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:42.172 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:19:42.196132+0000 2024-03-29T11:19:42.172 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:19:42.196132+0000 2024-03-29T11:19:42.172 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:42.172 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:19:42.172 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:19:42.172 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:19:42.173 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:19:42.173 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:19:42.173 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.196+0000 7fe44842a640 1 Processor -- start 2024-03-29T11:19:42.173 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.196+0000 7fe44842a640 1 -- start start 2024-03-29T11:19:42.174 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.196+0000 7fe44842a640 10 monclient: init 2024-03-29T11:19:42.174 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.197+0000 7fe44842a640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:42.174 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.197+0000 7fe44842a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:42.175 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.197+0000 7fe44842a640 10 monclient(hunting): picked mon.noname-b con 0x7fe4401457f0 addr v1:172.21.15.104:6790/0 2024-03-29T11:19:42.175 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.197+0000 7fe44842a640 10 monclient(hunting): picked mon.noname-c con 0x7fe44013f650 addr v1:172.21.15.119:6789/0 2024-03-29T11:19:42.175 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.197+0000 7fe44842a640 10 monclient(hunting): picked mon.noname-a con 0x7fe4400a9d70 addr v1:172.21.15.104:6789/0 2024-03-29T11:19:42.175 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.197+0000 7fe44842a640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4400aaef0 con 0x7fe4400a9d70 2024-03-29T11:19:42.175 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.197+0000 7fe44842a640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe440145bd0 con 0x7fe44013f650 2024-03-29T11:19:42.175 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.197+0000 7fe44842a640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe440145eb0 con 0x7fe4401457f0 2024-03-29T11:19:42.175 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.197+0000 7fe44842a640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:42.175 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.197+0000 7fe44842a640 10 monclient(hunting): authenticate will time out at 963.448402s 2024-03-29T11:19:42.175 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.197+0000 7fe4461c6640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7fe4401457f0 0x7fe44013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:42018/0 (socket says 172.21.15.104:42018) 2024-03-29T11:19:42.175 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.197+0000 7fe4461c6640 1 -- 172.21.15.104:0/1579084875 learned_addr learned my addr 172.21.15.104:0/1579084875 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:19:42.175 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.198+0000 7fe4451c4640 1 -- 172.21.15.104:0/1579084875 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 790355787 0 0) 0x7fe4400aaef0 con 0x7fe4400a9d70 2024-03-29T11:19:42.175 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.198+0000 7fe4451c4640 10 monclient(hunting): _negotiate 2024-03-29T11:19:42.176 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.198+0000 7fe4451c4640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:42.176 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.198+0000 7fe4451c4640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:42.176 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.198+0000 7fe4451c4640 10 monclient(hunting): my global_id is 4248 2024-03-29T11:19:42.176 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.198+0000 7fe4451c4640 1 -- 172.21.15.104:0/1579084875 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe428003520 con 0x7fe4400a9d70 2024-03-29T11:19:42.176 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.198+0000 7fe4451c4640 1 -- 172.21.15.104:0/1579084875 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3285866684 0 0) 0x7fe440145eb0 con 0x7fe4401457f0 2024-03-29T11:19:42.176 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.198+0000 7fe4451c4640 10 monclient(hunting): _negotiate 2024-03-29T11:19:42.176 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.198+0000 7fe4451c4640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:42.176 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.198+0000 7fe4451c4640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:42.176 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.198+0000 7fe4451c4640 10 monclient(hunting): my global_id is 4244 2024-03-29T11:19:42.176 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.198+0000 7fe4451c4640 1 -- 172.21.15.104:0/1579084875 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4400aaef0 con 0x7fe4401457f0 2024-03-29T11:19:42.179 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.199+0000 7fe4451c4640 1 -- 172.21.15.104:0/1579084875 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1283768353 0 0) 0x7fe4400aaef0 con 0x7fe4401457f0 2024-03-29T11:19:42.179 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.199+0000 7fe4451c4640 1 -- 172.21.15.104:0/1579084875 >> v1:172.21.15.119:6789/0 conn(0x7fe44013f650 legacy=0x7fe4400b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:42.179 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.199+0000 7fe4451c4640 1 -- 172.21.15.104:0/1579084875 >> v1:172.21.15.104:6789/0 conn(0x7fe4400a9d70 legacy=0x7fe44014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:42.179 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.199+0000 7fe4451c4640 10 monclient: _finish_hunting 0 2024-03-29T11:19:42.179 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.199+0000 7fe4451c4640 1 monclient: found mon.noname-b 2024-03-29T11:19:42.179 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.199+0000 7fe4451c4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:42.179 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.199+0000 7fe4451c4640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.104:6790/0 2024-03-29T11:19:42.179 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.199+0000 7fe4451c4640 1 -- 172.21.15.104:0/1579084875 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe440146340 con 0x7fe4401457f0 2024-03-29T11:19:42.179 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.199+0000 7fe4451c4640 10 monclient: _finish_auth 0 2024-03-29T11:19:42.179 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.199+0000 7fe4451c4640 10 monclient: _check_auth_tickets 2024-03-29T11:19:42.179 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.199+0000 7fe4451c4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:42.179 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.199+0000 7fe44842a640 5 monclient: authenticate success, global_id 4244 2024-03-29T11:19:42.179 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.199+0000 7fe44842a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:42.180 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.199+0000 7fe4451c4640 1 -- 172.21.15.104:0/1579084875 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fe430001630 con 0x7fe4401457f0 2024-03-29T11:19:42.180 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.199+0000 7fe4451c4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:42.180 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.199+0000 7fe4451c4640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:42.180 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.199+0000 7fe4451c4640 10 monclient: dump: 2024-03-29T11:19:42.180 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:19:42.180 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:42.180 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:42.180 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:42.180 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:42.180 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:19:42.180 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:42.180 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:42.180 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:42.180 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:19:42.180 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.199+0000 7fe44842a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:42.181 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.199+0000 7fe4451c4640 1 -- 172.21.15.104:0/1579084875 <== mon.2 v1:172.21.15.104:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe430001880 con 0x7fe4401457f0 2024-03-29T11:19:42.181 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.199+0000 7fe4451c4640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:42.181 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.199+0000 7fe44842a640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:42.181 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.199+0000 7fe44842a640 4 set_mon_vals no callback set 2024-03-29T11:19:42.181 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.200+0000 7fe44842a640 10 monclient: shutdown 2024-03-29T11:19:42.181 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.200+0000 7fe44842a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:42.181 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.200+0000 7fe44842a640 1 -- 172.21.15.104:0/1579084875 >> v1:172.21.15.104:6790/0 conn(0x7fe4401457f0 legacy=0x7fe44013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:42.181 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.200+0000 7fe44842a640 1 -- 172.21.15.104:0/1579084875 shutdown_connections 2024-03-29T11:19:42.181 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.200+0000 7fe44842a640 1 -- 172.21.15.104:0/1579084875 >> 172.21.15.104:0/1579084875 conn(0x7fe440066a00 msgr2=0x7fe440066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:42.181 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.201+0000 7fe44842a640 1 -- 172.21.15.104:0/1579084875 shutdown_connections 2024-03-29T11:19:42.181 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.201+0000 7fe44842a640 1 -- 172.21.15.104:0/1579084875 wait complete. 2024-03-29T11:19:42.181 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.201+0000 7fe44842a640 10 monclient: build_initial_monmap 2024-03-29T11:19:42.181 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.201+0000 7fe44842a640 10 monclient: monmap: 2024-03-29T11:19:42.181 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:19:42.181 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:42.182 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:19:42.182 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:19:42.182 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:42.182 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:19:42.182 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:19:42.182 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:19:42.182 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:19:42.182 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:19:42.182 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.202+0000 7fe44842a640 1 librados: starting msgr at 2024-03-29T11:19:42.182 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.202+0000 7fe44842a640 1 librados: starting objecter 2024-03-29T11:19:42.182 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.202+0000 7fe44842a640 1 Processor -- start 2024-03-29T11:19:42.182 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.202+0000 7fe44842a640 1 -- start start 2024-03-29T11:19:42.182 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.202+0000 7fe44842a640 1 librados: setting wanted keys 2024-03-29T11:19:42.182 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.202+0000 7fe44842a640 1 librados: calling monclient init 2024-03-29T11:19:42.182 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.202+0000 7fe44842a640 10 monclient: init 2024-03-29T11:19:42.187 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.203+0000 7fe44842a640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:42.187 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.203+0000 7fe44842a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:42.187 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.203+0000 7fe44842a640 10 monclient(hunting): picked mon.noname-c con 0x7fe4400a9d70 addr v1:172.21.15.104:6790/0 2024-03-29T11:19:42.187 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.203+0000 7fe44842a640 10 monclient(hunting): picked mon.noname-b con 0x7fe44013f650 addr v1:172.21.15.119:6789/0 2024-03-29T11:19:42.188 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.203+0000 7fe44842a640 10 monclient(hunting): picked mon.noname-a con 0x7fe4401457f0 addr v1:172.21.15.104:6789/0 2024-03-29T11:19:42.188 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.203+0000 7fe44842a640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4400b0930 con 0x7fe4401457f0 2024-03-29T11:19:42.188 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.203+0000 7fe44842a640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4400ac110 con 0x7fe44013f650 2024-03-29T11:19:42.188 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.203+0000 7fe44842a640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4400ac3f0 con 0x7fe4400a9d70 2024-03-29T11:19:42.188 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.203+0000 7fe44842a640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:42.188 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.203+0000 7fe44842a640 10 monclient(hunting): authenticate will time out at 963.454705s 2024-03-29T11:19:42.188 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.203+0000 7fe4461c6640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7fe4400a9d70 0x7fe440071730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:43196/0 (socket says 172.21.15.104:43196) 2024-03-29T11:19:42.188 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.203+0000 7fe4461c6640 1 -- 172.21.15.104:0/936346305 learned_addr learned my addr 172.21.15.104:0/936346305 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:19:42.188 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.203+0000 7fe43effd640 10 client.?.objecter ms_handle_connect 0x7fe4400a9d70 2024-03-29T11:19:42.188 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.203+0000 7fe43effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:42.188 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.204+0000 7fe43effd640 1 -- 172.21.15.104:0/936346305 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2853462286 0 0) 0x7fe4400ac3f0 con 0x7fe4400a9d70 2024-03-29T11:19:42.188 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.204+0000 7fe43effd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:42.188 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.204+0000 7fe43effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:42.189 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.204+0000 7fe43effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:42.189 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.204+0000 7fe43effd640 10 monclient(hunting): my global_id is 4247 2024-03-29T11:19:42.189 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.204+0000 7fe43effd640 1 -- 172.21.15.104:0/936346305 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe42c004eb0 con 0x7fe4400a9d70 2024-03-29T11:19:42.189 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.204+0000 7fe43effd640 10 client.?.objecter ms_handle_connect 0x7fe44013f650 2024-03-29T11:19:42.189 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.204+0000 7fe43effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:42.189 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.204+0000 7fe43effd640 1 -- 172.21.15.104:0/936346305 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1401153327 0 0) 0x7fe42c004eb0 con 0x7fe4400a9d70 2024-03-29T11:19:42.189 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.204+0000 7fe43effd640 1 -- 172.21.15.104:0/936346305 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4400ac3f0 con 0x7fe4400a9d70 2024-03-29T11:19:42.189 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.204+0000 7fe43effd640 1 -- 172.21.15.104:0/936346305 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fe430004980 con 0x7fe4400a9d70 2024-03-29T11:19:42.189 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.204+0000 7fe43effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:42.189 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.204+0000 7fe43effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:42.189 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.204+0000 7fe43effd640 10 monclient(hunting): dump: 2024-03-29T11:19:42.189 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:19:42.189 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:42.190 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:42.190 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:42.190 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:42.190 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:19:42.190 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:42.190 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:42.190 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:42.190 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:19:42.190 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.204+0000 7fe43effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:42.190 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.205+0000 7fe43effd640 1 -- 172.21.15.104:0/936346305 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2113952 0 0) 0x7fe4400ac110 con 0x7fe44013f650 2024-03-29T11:19:42.190 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.205+0000 7fe43effd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:42.190 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.205+0000 7fe43effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:42.190 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.205+0000 7fe43effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:42.190 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.205+0000 7fe43effd640 10 monclient(hunting): my global_id is 4231 2024-03-29T11:19:42.190 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.205+0000 7fe43effd640 1 -- 172.21.15.104:0/936346305 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe42c004eb0 con 0x7fe44013f650 2024-03-29T11:19:42.191 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.205+0000 7fe43effd640 10 client.?.objecter ms_handle_connect 0x7fe4401457f0 2024-03-29T11:19:42.191 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.205+0000 7fe43effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:42.191 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.205+0000 7fe43effd640 1 -- 172.21.15.104:0/936346305 <== mon.2 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3779246235 0 0) 0x7fe4400ac3f0 con 0x7fe4400a9d70 2024-03-29T11:19:42.191 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.205+0000 7fe43effd640 1 -- 172.21.15.104:0/936346305 >> v1:172.21.15.119:6789/0 conn(0x7fe44013f650 legacy=0x7fe440070c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:42.191 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.205+0000 7fe43effd640 1 -- 172.21.15.104:0/936346305 >> v1:172.21.15.104:6789/0 conn(0x7fe4401457f0 legacy=0x7fe4401e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:42.191 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.205+0000 7fe43effd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:42.191 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.205+0000 7fe43effd640 1 monclient: found mon.c 2024-03-29T11:19:42.191 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.205+0000 7fe43effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:42.191 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.205+0000 7fe43effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:19:42.191 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.205+0000 7fe43effd640 1 -- 172.21.15.104:0/936346305 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4400ac6d0 con 0x7fe4400a9d70 2024-03-29T11:19:42.191 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.205+0000 7fe43effd640 10 monclient: _finish_auth 0 2024-03-29T11:19:42.191 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.205+0000 7fe43effd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:42.191 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.205+0000 7fe43effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:42.191 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.205+0000 7fe44842a640 5 monclient: authenticate success, global_id 4247 2024-03-29T11:19:42.191 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.205+0000 7fe44842a640 10 monclient: _renew_subs 2024-03-29T11:19:42.192 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.205+0000 7fe44842a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:19:42.192 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.205+0000 7fe44842a640 1 -- 172.21.15.104:0/936346305 --> v1:172.21.15.104:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4400b13c0 con 0x7fe4400a9d70 2024-03-29T11:19:42.192 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.205+0000 7fe44842a640 10 client.4247.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:42.192 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.205+0000 7fe44842a640 10 monclient: _renew_subs 2024-03-29T11:19:42.192 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.205+0000 7fe44842a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:19:42.192 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.205+0000 7fe44842a640 1 -- 172.21.15.104:0/936346305 --> v1:172.21.15.104:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4400713a0 con 0x7fe4400a9d70 2024-03-29T11:19:42.192 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.205+0000 7fe44842a640 1 librados: init done 2024-03-29T11:19:42.192 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.206+0000 7fe43effd640 1 -- 172.21.15.104:0/936346305 <== mon.2 v1:172.21.15.104:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe430004ec0 con 0x7fe4400a9d70 2024-03-29T11:19:42.192 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.206+0000 7fe43effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:42.192 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.206+0000 7fe43effd640 1 -- 172.21.15.104:0/936346305 <== mon.2 v1:172.21.15.104:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fe430005290 con 0x7fe4400a9d70 2024-03-29T11:19:42.192 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.206+0000 7fe4449c3640 4 set_mon_vals no callback set 2024-03-29T11:19:42.192 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.206+0000 7fe43effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:42.192 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.206+0000 7fe43effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:42.192 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.206+0000 7fe43effd640 10 monclient: dump: 2024-03-29T11:19:42.192 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:19:42.193 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:42.193 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:42.193 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:42.193 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:42.193 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:19:42.193 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:42.193 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:42.193 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:42.193 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:19:42.193 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.207+0000 7fe44842a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:42.193 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.207+0000 7fe44842a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:42.193 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.207+0000 7fe44842a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:19:42.193 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.207+0000 7fe44842a640 1 -- 172.21.15.104:0/936346305 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe420005080 con 0x7fe4400a9d70 2024-03-29T11:19:42.193 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.208+0000 7fe43effd640 1 -- 172.21.15.104:0/936346305 <== mon.2 v1:172.21.15.104:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7fe430001490 con 0x7fe4400a9d70 2024-03-29T11:19:42.193 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.209+0000 7fe4459c5640 10 monclient: get_auth_request con 0x7fe42c06dd30 auth_method 0 2024-03-29T11:19:42.194 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.209+0000 7fe43effd640 1 -- 172.21.15.104:0/936346305 <== mon.2 v1:172.21.15.104:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7fe430084c50 con 0x7fe4400a9d70 2024-03-29T11:19:42.194 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.209+0000 7fe43effd640 10 client.4247.objecter ms_dispatch 0x7fe440000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:42.194 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.209+0000 7fe43effd640 3 client.4247.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:42.194 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.209+0000 7fe43effd640 3 client.4247.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:42.194 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.209+0000 7fe43effd640 20 client.4247.objecter dump_active .. 0 homeless 2024-03-29T11:19:42.194 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.209+0000 7fe43effd640 10 client.4247.objecter ms_handle_connect 0x7fe42c06dd30 2024-03-29T11:19:42.194 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.218+0000 7fe43effd640 1 -- 172.21.15.104:0/936346305 <== mon.2 v1:172.21.15.104:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe430050530 con 0x7fe4400a9d70 2024-03-29T11:19:42.194 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.218+0000 7fe43effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:42.194 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.218+0000 7fe43effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:42.516 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.539+0000 7fe44842a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:42.516 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.539+0000 7fe44842a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:42.516 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.539+0000 7fe44842a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:19:42.517 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.539+0000 7fe44842a640 1 -- 172.21.15.104:0/936346305 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe420005330 con 0x7fe4400a9d70 2024-03-29T11:19:42.517 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.540+0000 7fe43effd640 1 -- 172.21.15.104:0/936346305 <== mon.2 v1:172.21.15.104:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7fe430052460 con 0x7fe4400a9d70 2024-03-29T11:19:42.517 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.540+0000 7fe43effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:42.517 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.540+0000 7fe43effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:42.518 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:19:42.518 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:42.521 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.544+0000 7fe44842a640 10 librados: watch_flush enter 2024-03-29T11:19:42.521 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.544+0000 7fe44842a640 10 librados: watch_flush exit 2024-03-29T11:19:42.521 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.544+0000 7fe44842a640 20 client.4247.objecter shutdown clearing up homeless session... 2024-03-29T11:19:42.521 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.544+0000 7fe44842a640 10 client.4247.objecter successfully canceled tick 2024-03-29T11:19:42.521 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.544+0000 7fe44842a640 1 -- 172.21.15.104:0/936346305 >> v1:172.21.15.104:6816/3341126660 conn(0x7fe42c06dd30 legacy=0x7fe42c0701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:42.521 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.544+0000 7fe44842a640 10 monclient: shutdown 2024-03-29T11:19:42.521 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.544+0000 7fe44842a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:42.521 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.544+0000 7fe44842a640 1 -- 172.21.15.104:0/936346305 >> v1:172.21.15.104:6790/0 conn(0x7fe4400a9d70 legacy=0x7fe440071730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:42.521 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.544+0000 7fe44842a640 1 -- 172.21.15.104:0/936346305 shutdown_connections 2024-03-29T11:19:42.521 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.544+0000 7fe44842a640 1 -- 172.21.15.104:0/936346305 >> 172.21.15.104:0/936346305 conn(0x7fe440066a00 msgr2=0x7fe4400b2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:42.522 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.545+0000 7fe44842a640 1 -- 172.21.15.104:0/936346305 shutdown_connections 2024-03-29T11:19:42.522 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.545+0000 7fe44842a640 1 -- 172.21.15.104:0/936346305 wait complete. 2024-03-29T11:19:42.522 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:42.545+0000 7fe44842a640 1 librados: shutdown 2024-03-29T11:19:42.536 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:49.343 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:49.486 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.509+0000 7f37481f4640 10 monclient: get_monmap_and_config 2024-03-29T11:19:49.486 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.509+0000 7f37481f4640 10 monclient: build_initial_monmap 2024-03-29T11:19:49.487 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.509+0000 7f37481f4640 10 monclient: monmap: 2024-03-29T11:19:49.487 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:19:49.487 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:49.487 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:19:49.510655+0000 2024-03-29T11:19:49.487 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:19:49.510655+0000 2024-03-29T11:19:49.487 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:49.487 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:19:49.487 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:19:49.487 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:19:49.487 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:19:49.487 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:19:49.488 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.511+0000 7f37481f4640 1 Processor -- start 2024-03-29T11:19:49.488 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.511+0000 7f37481f4640 1 -- start start 2024-03-29T11:19:49.488 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.511+0000 7f37481f4640 10 monclient: init 2024-03-29T11:19:49.490 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.512+0000 7f37481f4640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:49.490 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.512+0000 7f37481f4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:49.490 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.512+0000 7f37481f4640 10 monclient(hunting): picked mon.noname-c con 0x7f37400708a0 addr v1:172.21.15.119:6789/0 2024-03-29T11:19:49.490 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.512+0000 7f37481f4640 10 monclient(hunting): picked mon.noname-b con 0x7f374006fe90 addr v1:172.21.15.104:6790/0 2024-03-29T11:19:49.490 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.512+0000 7f37481f4640 10 monclient(hunting): picked mon.noname-a con 0x7f3740143c50 addr v1:172.21.15.104:6789/0 2024-03-29T11:19:49.490 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.512+0000 7f37481f4640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3740072630 con 0x7f3740143c50 2024-03-29T11:19:49.490 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.512+0000 7f37481f4640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3740071410 con 0x7f37400708a0 2024-03-29T11:19:49.490 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.512+0000 7f37481f4640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37400afaa0 con 0x7f374006fe90 2024-03-29T11:19:49.490 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.512+0000 7f37481f4640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:49.490 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.512+0000 7f37481f4640 10 monclient(hunting): authenticate will time out at 970.763657s 2024-03-29T11:19:49.490 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.513+0000 7f3745f90640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f37400708a0 0x7f37400af390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:39374/0 (socket says 172.21.15.104:39374) 2024-03-29T11:19:49.491 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.513+0000 7f3745f90640 1 -- 172.21.15.104:0/2444252228 learned_addr learned my addr 172.21.15.104:0/2444252228 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:19:49.492 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.515+0000 7f3744f8e640 1 -- 172.21.15.104:0/2444252228 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1972345709 0 0) 0x7f3740071410 con 0x7f37400708a0 2024-03-29T11:19:49.492 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.515+0000 7f3744f8e640 10 monclient(hunting): _negotiate 2024-03-29T11:19:49.492 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.515+0000 7f3744f8e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:49.492 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.515+0000 7f3744f8e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:49.492 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.515+0000 7f3744f8e640 10 monclient(hunting): my global_id is 4234 2024-03-29T11:19:49.492 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.515+0000 7f3744f8e640 1 -- 172.21.15.104:0/2444252228 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f372c003520 con 0x7f37400708a0 2024-03-29T11:19:49.493 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.516+0000 7f3744f8e640 1 -- 172.21.15.104:0/2444252228 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3838023721 0 0) 0x7f372c003520 con 0x7f37400708a0 2024-03-29T11:19:49.493 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.516+0000 7f3744f8e640 1 -- 172.21.15.104:0/2444252228 >> v1:172.21.15.104:6790/0 conn(0x7f374006fe90 legacy=0x7f3740143540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:19:49.493 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.516+0000 7f3744f8e640 1 -- 172.21.15.104:0/2444252228 >> v1:172.21.15.104:6789/0 conn(0x7f3740143c50 legacy=0x7f374014b8e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:19:49.494 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.517+0000 7f3744f8e640 10 monclient: _finish_hunting 0 2024-03-29T11:19:49.495 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.517+0000 7f3744f8e640 1 monclient: found mon.noname-c 2024-03-29T11:19:49.495 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.517+0000 7f3744f8e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:49.495 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.517+0000 7f3744f8e640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.119:6789/0 2024-03-29T11:19:49.495 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.517+0000 7f3744f8e640 1 -- 172.21.15.104:0/2444252228 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f374006dca0 con 0x7f37400708a0 2024-03-29T11:19:49.495 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.517+0000 7f3744f8e640 10 monclient: _finish_auth 0 2024-03-29T11:19:49.495 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.517+0000 7f3744f8e640 10 monclient: _check_auth_tickets 2024-03-29T11:19:49.495 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.517+0000 7f3744f8e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:49.495 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.517+0000 7f37481f4640 5 monclient: authenticate success, global_id 4234 2024-03-29T11:19:49.495 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.517+0000 7f3744f8e640 1 -- 172.21.15.104:0/2444252228 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f373c001690 con 0x7f37400708a0 2024-03-29T11:19:49.495 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.517+0000 7f37481f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:49.495 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.517+0000 7f3744f8e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:49.495 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.517+0000 7f3744f8e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:49.495 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.517+0000 7f3744f8e640 10 monclient: dump: 2024-03-29T11:19:49.495 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:19:49.496 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:49.496 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:49.496 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:49.496 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:49.496 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:19:49.496 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:49.496 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:49.496 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:49.496 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:19:49.496 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.517+0000 7f3744f8e640 1 -- 172.21.15.104:0/2444252228 <== mon.1 v1:172.21.15.119:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f373c0018b0 con 0x7f37400708a0 2024-03-29T11:19:49.496 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.517+0000 7f37481f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:49.496 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.517+0000 7f3744f8e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:49.496 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.517+0000 7f37481f4640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:49.496 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.517+0000 7f3744f8e640 1 -- 172.21.15.104:0/2444252228 <== mon.1 v1:172.21.15.119:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f373c001c40 con 0x7f37400708a0 2024-03-29T11:19:49.496 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.517+0000 7f3744f8e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:49.497 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.517+0000 7f37481f4640 4 set_mon_vals no callback set 2024-03-29T11:19:49.497 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.517+0000 7f3744f8e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:49.497 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.517+0000 7f3744f8e640 10 monclient: dump: 2024-03-29T11:19:49.497 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:19:49.497 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:49.497 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:49.497 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:49.497 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:49.497 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:19:49.497 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:49.497 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:49.497 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:49.497 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:19:49.499 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.518+0000 7f37481f4640 10 monclient: shutdown 2024-03-29T11:19:49.499 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.518+0000 7f37481f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:49.499 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.518+0000 7f37481f4640 1 -- 172.21.15.104:0/2444252228 >> v1:172.21.15.119:6789/0 conn(0x7f37400708a0 legacy=0x7f37400af390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:49.499 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.519+0000 7f37481f4640 1 -- 172.21.15.104:0/2444252228 shutdown_connections 2024-03-29T11:19:49.499 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.519+0000 7f37481f4640 1 -- 172.21.15.104:0/2444252228 >> 172.21.15.104:0/2444252228 conn(0x7f37400675b0 msgr2=0x7f37400b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:49.499 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.519+0000 7f37481f4640 1 -- 172.21.15.104:0/2444252228 shutdown_connections 2024-03-29T11:19:49.499 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.519+0000 7f37481f4640 1 -- 172.21.15.104:0/2444252228 wait complete. 2024-03-29T11:19:49.499 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.520+0000 7f37481f4640 10 monclient: build_initial_monmap 2024-03-29T11:19:49.499 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.520+0000 7f37481f4640 10 monclient: monmap: 2024-03-29T11:19:49.500 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:19:49.500 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:49.500 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:19:49.500 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:19:49.500 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:49.500 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:19:49.500 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:19:49.500 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:19:49.500 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:19:49.500 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:19:49.500 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.521+0000 7f37481f4640 1 librados: starting msgr at 2024-03-29T11:19:49.500 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.521+0000 7f37481f4640 1 librados: starting objecter 2024-03-29T11:19:49.500 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.521+0000 7f37481f4640 1 Processor -- start 2024-03-29T11:19:49.500 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.521+0000 7f37481f4640 1 -- start start 2024-03-29T11:19:49.500 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.521+0000 7f37481f4640 1 librados: setting wanted keys 2024-03-29T11:19:49.501 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.521+0000 7f37481f4640 1 librados: calling monclient init 2024-03-29T11:19:49.501 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.521+0000 7f37481f4640 10 monclient: init 2024-03-29T11:19:49.501 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.522+0000 7f37481f4640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:49.501 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.522+0000 7f37481f4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:49.501 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.522+0000 7f37481f4640 10 monclient(hunting): picked mon.noname-c con 0x7f374006fe90 addr v1:172.21.15.104:6790/0 2024-03-29T11:19:49.501 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.522+0000 7f37481f4640 10 monclient(hunting): picked mon.noname-b con 0x7f37400708a0 addr v1:172.21.15.119:6789/0 2024-03-29T11:19:49.502 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.522+0000 7f374578f640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f37400708a0 0x7f37400abeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:39470/0 (socket says 172.21.15.104:39470) 2024-03-29T11:19:49.502 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.523+0000 7f37481f4640 10 monclient(hunting): picked mon.noname-a con 0x7f3740143c50 addr v1:172.21.15.104:6789/0 2024-03-29T11:19:49.502 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.523+0000 7f374578f640 1 -- 172.21.15.104:0/2273306886 learned_addr learned my addr 172.21.15.104:0/2273306886 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:19:49.502 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.523+0000 7f37481f4640 1 -- 172.21.15.104:0/2273306886 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3740140800 con 0x7f3740143c50 2024-03-29T11:19:49.502 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.523+0000 7f37481f4640 1 -- 172.21.15.104:0/2273306886 --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3740140aa0 con 0x7f37400708a0 2024-03-29T11:19:49.502 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.523+0000 7f37481f4640 1 -- 172.21.15.104:0/2273306886 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37400aa0a0 con 0x7f374006fe90 2024-03-29T11:19:49.502 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.523+0000 7f37481f4640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:49.502 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.523+0000 7f37481f4640 10 monclient(hunting): authenticate will time out at 970.774353s 2024-03-29T11:19:49.502 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.524+0000 7f37367fc640 10 client.?.objecter ms_handle_connect 0x7f37400708a0 2024-03-29T11:19:49.502 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.524+0000 7f37367fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:49.502 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.524+0000 7f37367fc640 1 -- 172.21.15.104:0/2273306886 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2756707161 0 0) 0x7f3740140aa0 con 0x7f37400708a0 2024-03-29T11:19:49.502 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.524+0000 7f37367fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:49.503 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.524+0000 7f37367fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:49.503 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.524+0000 7f37367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:49.503 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.524+0000 7f37367fc640 10 monclient(hunting): my global_id is 4237 2024-03-29T11:19:49.503 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.524+0000 7f37367fc640 1 -- 172.21.15.104:0/2273306886 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3724004eb0 con 0x7f37400708a0 2024-03-29T11:19:49.504 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.525+0000 7f37367fc640 10 client.?.objecter ms_handle_connect 0x7f374006fe90 2024-03-29T11:19:49.504 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.525+0000 7f37367fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:49.504 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.525+0000 7f37367fc640 1 -- 172.21.15.104:0/2273306886 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3429131011 0 0) 0x7f3724004eb0 con 0x7f37400708a0 2024-03-29T11:19:49.504 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.525+0000 7f37367fc640 1 -- 172.21.15.104:0/2273306886 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3740140aa0 con 0x7f37400708a0 2024-03-29T11:19:49.504 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.526+0000 7f37367fc640 1 -- 172.21.15.104:0/2273306886 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 721163801 0 0) 0x7f37400aa0a0 con 0x7f374006fe90 2024-03-29T11:19:49.505 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.526+0000 7f37367fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:49.505 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.526+0000 7f37367fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:49.505 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.526+0000 7f37367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:49.505 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.526+0000 7f37367fc640 10 monclient(hunting): my global_id is 4250 2024-03-29T11:19:49.505 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.526+0000 7f37367fc640 1 -- 172.21.15.104:0/2273306886 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3724004eb0 con 0x7f374006fe90 2024-03-29T11:19:49.505 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.526+0000 7f37367fc640 1 -- 172.21.15.104:0/2273306886 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f3738001b50 con 0x7f37400708a0 2024-03-29T11:19:49.505 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.526+0000 7f37367fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:49.505 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.526+0000 7f37367fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:49.505 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.526+0000 7f37367fc640 10 monclient(hunting): dump: 2024-03-29T11:19:49.505 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:19:49.505 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:49.505 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:49.505 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:49.505 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:49.505 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:19:49.505 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:49.506 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:49.506 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:49.506 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:19:49.506 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.526+0000 7f37367fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:49.506 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.526+0000 7f37367fc640 1 -- 172.21.15.104:0/2273306886 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2463984326 0 0) 0x7f3724004eb0 con 0x7f374006fe90 2024-03-29T11:19:49.506 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.526+0000 7f37367fc640 1 -- 172.21.15.104:0/2273306886 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37400aa0a0 con 0x7f374006fe90 2024-03-29T11:19:49.506 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.526+0000 7f37367fc640 1 -- 172.21.15.104:0/2273306886 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f373c0013b0 con 0x7f374006fe90 2024-03-29T11:19:49.506 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.526+0000 7f37367fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:49.506 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.526+0000 7f37367fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:49.506 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.526+0000 7f37367fc640 10 monclient(hunting): dump: 2024-03-29T11:19:49.506 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:19:49.506 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:49.506 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:49.506 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:49.506 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:49.507 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:19:49.507 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:49.507 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:49.507 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:49.507 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:19:49.507 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.526+0000 7f37367fc640 1 -- 172.21.15.104:0/2273306886 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1993563618 0 0) 0x7f3740140aa0 con 0x7f37400708a0 2024-03-29T11:19:49.507 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.527+0000 7f37367fc640 1 -- 172.21.15.104:0/2273306886 >> v1:172.21.15.104:6790/0 conn(0x7f374006fe90 legacy=0x7f37400ab7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:49.507 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.527+0000 7f37367fc640 1 -- 172.21.15.104:0/2273306886 >> v1:172.21.15.104:6789/0 conn(0x7f3740143c50 legacy=0x7f37401ee7e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:19:49.507 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.530+0000 7f37367fc640 10 monclient: _finish_hunting 0 2024-03-29T11:19:49.508 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.531+0000 7f37367fc640 1 monclient: found mon.b 2024-03-29T11:19:49.508 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.531+0000 7f37367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:49.508 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.531+0000 7f37367fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:19:49.508 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.531+0000 7f37367fc640 1 -- 172.21.15.104:0/2273306886 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37400aa230 con 0x7f37400708a0 2024-03-29T11:19:49.508 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.531+0000 7f37367fc640 10 monclient: _finish_auth 0 2024-03-29T11:19:49.508 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.531+0000 7f37367fc640 10 monclient: _check_auth_tickets 2024-03-29T11:19:49.508 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.531+0000 7f37367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:49.508 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.531+0000 7f37481f4640 5 monclient: authenticate success, global_id 4237 2024-03-29T11:19:49.508 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.531+0000 7f37481f4640 10 monclient: _renew_subs 2024-03-29T11:19:49.508 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.531+0000 7f37481f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:19:49.508 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.531+0000 7f37481f4640 1 -- 172.21.15.104:0/2273306886 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37400a9600 con 0x7f37400708a0 2024-03-29T11:19:49.508 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.531+0000 7f37481f4640 10 client.4237.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:49.508 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.531+0000 7f37481f4640 10 monclient: _renew_subs 2024-03-29T11:19:49.508 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.531+0000 7f37481f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:19:49.508 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.531+0000 7f37481f4640 1 -- 172.21.15.104:0/2273306886 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f37400b54b0 con 0x7f37400708a0 2024-03-29T11:19:49.509 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.531+0000 7f37481f4640 1 librados: init done 2024-03-29T11:19:49.510 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.533+0000 7f37367fc640 1 -- 172.21.15.104:0/2273306886 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3738001cb0 con 0x7f37400708a0 2024-03-29T11:19:49.510 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.533+0000 7f37367fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:49.510 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.533+0000 7f37367fc640 1 -- 172.21.15.104:0/2273306886 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f3738004d80 con 0x7f37400708a0 2024-03-29T11:19:49.510 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.533+0000 7f37367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:49.510 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.533+0000 7f37367fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:49.510 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.533+0000 7f3737fff640 4 set_mon_vals no callback set 2024-03-29T11:19:49.510 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.533+0000 7f37367fc640 10 monclient: dump: 2024-03-29T11:19:49.510 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:19:49.510 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:49.511 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:49.511 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:49.511 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:49.511 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:19:49.511 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:49.511 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:49.511 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:49.511 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:19:49.511 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.534+0000 7f37481f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:49.512 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.534+0000 7f37481f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:49.512 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.535+0000 7f37481f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:19:49.512 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.535+0000 7f37481f4640 1 -- 172.21.15.104:0/2273306886 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3728005080 con 0x7f37400708a0 2024-03-29T11:19:49.512 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.535+0000 7f37367fc640 1 -- 172.21.15.104:0/2273306886 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f373801ad50 con 0x7f37400708a0 2024-03-29T11:19:49.513 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.536+0000 7f37367fc640 1 -- 172.21.15.104:0/2273306886 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f3738084e20 con 0x7f37400708a0 2024-03-29T11:19:49.513 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.536+0000 7f37367fc640 10 client.4237.objecter ms_dispatch 0x7f3740000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:49.513 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.536+0000 7f37367fc640 3 client.4237.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:49.513 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.536+0000 7f37367fc640 3 client.4237.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:49.513 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.536+0000 7f3745f90640 10 monclient: get_auth_request con 0x7f372406de40 auth_method 0 2024-03-29T11:19:49.514 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.537+0000 7f37367fc640 20 client.4237.objecter dump_active .. 0 homeless 2024-03-29T11:19:49.514 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.537+0000 7f37367fc640 10 client.4237.objecter ms_handle_connect 0x7f372406de40 2024-03-29T11:19:49.520 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.543+0000 7f37367fc640 1 -- 172.21.15.104:0/2273306886 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3738050530 con 0x7f37400708a0 2024-03-29T11:19:49.520 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.544+0000 7f37367fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:49.520 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.544+0000 7f37367fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:49.842 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.865+0000 7f37481f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:49.842 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.865+0000 7f37481f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:49.842 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.865+0000 7f37481f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:19:49.843 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.865+0000 7f37481f4640 1 -- 172.21.15.104:0/2273306886 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3728005330 con 0x7f37400708a0 2024-03-29T11:19:49.844 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.867+0000 7f37367fc640 1 -- 172.21.15.104:0/2273306886 <== mon.1 v1:172.21.15.119:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f3738052630 con 0x7f37400708a0 2024-03-29T11:19:49.844 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.867+0000 7f37367fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:49.844 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.867+0000 7f37367fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:49.844 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:19:49.844 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:49.847 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.871+0000 7f37481f4640 10 librados: watch_flush enter 2024-03-29T11:19:49.848 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.871+0000 7f37481f4640 10 librados: watch_flush exit 2024-03-29T11:19:49.848 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.871+0000 7f37481f4640 20 client.4237.objecter shutdown clearing up homeless session... 2024-03-29T11:19:49.848 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.871+0000 7f37481f4640 10 client.4237.objecter successfully canceled tick 2024-03-29T11:19:49.849 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.871+0000 7f37481f4640 1 -- 172.21.15.104:0/2273306886 >> v1:172.21.15.104:6816/3341126660 conn(0x7f372406de40 legacy=0x7f3724070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:49.849 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.872+0000 7f37481f4640 10 monclient: shutdown 2024-03-29T11:19:49.849 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.872+0000 7f37481f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:49.849 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.872+0000 7f37481f4640 1 -- 172.21.15.104:0/2273306886 >> v1:172.21.15.119:6789/0 conn(0x7f37400708a0 legacy=0x7f37400abeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:49.849 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.872+0000 7f37481f4640 1 -- 172.21.15.104:0/2273306886 shutdown_connections 2024-03-29T11:19:49.849 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.872+0000 7f37481f4640 1 -- 172.21.15.104:0/2273306886 >> 172.21.15.104:0/2273306886 conn(0x7f37400675b0 msgr2=0x7f37400b0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:49.849 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.872+0000 7f37481f4640 1 -- 172.21.15.104:0/2273306886 shutdown_connections 2024-03-29T11:19:49.849 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.872+0000 7f37481f4640 1 -- 172.21.15.104:0/2273306886 wait complete. 2024-03-29T11:19:49.849 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:49.872+0000 7f37481f4640 1 librados: shutdown 2024-03-29T11:19:49.864 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:56.671 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:56.806 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.829+0000 7fb72fe33640 10 monclient: get_monmap_and_config 2024-03-29T11:19:56.806 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.829+0000 7fb72fe33640 10 monclient: build_initial_monmap 2024-03-29T11:19:56.806 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.829+0000 7fb72fe33640 10 monclient: monmap: 2024-03-29T11:19:56.806 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:19:56.806 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:56.806 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:19:56.830086+0000 2024-03-29T11:19:56.806 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:19:56.830086+0000 2024-03-29T11:19:56.806 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:56.806 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:19:56.806 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:19:56.806 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:19:56.807 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:19:56.807 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:19:56.807 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.830+0000 7fb72fe33640 1 Processor -- start 2024-03-29T11:19:56.807 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.830+0000 7fb72fe33640 1 -- start start 2024-03-29T11:19:56.808 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.830+0000 7fb72fe33640 10 monclient: init 2024-03-29T11:19:56.808 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.831+0000 7fb72fe33640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:56.808 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.831+0000 7fb72fe33640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:56.808 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.831+0000 7fb72fe33640 10 monclient(hunting): picked mon.noname-b con 0x7fb7281457f0 addr v1:172.21.15.104:6790/0 2024-03-29T11:19:56.808 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.831+0000 7fb72fe33640 10 monclient(hunting): picked mon.noname-c con 0x7fb72813f650 addr v1:172.21.15.119:6789/0 2024-03-29T11:19:56.810 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.834+0000 7fb72fe33640 10 monclient(hunting): picked mon.noname-a con 0x7fb7280a9d70 addr v1:172.21.15.104:6789/0 2024-03-29T11:19:56.811 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.834+0000 7fb72fe33640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7280aaef0 con 0x7fb7280a9d70 2024-03-29T11:19:56.811 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.834+0000 7fb72fe33640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb728145bd0 con 0x7fb72813f650 2024-03-29T11:19:56.811 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.835+0000 7fb72fe33640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb728145eb0 con 0x7fb7281457f0 2024-03-29T11:19:56.812 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.835+0000 7fb72fe33640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:56.812 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.835+0000 7fb72e3d0640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fb7280a9d70 0x7fb72814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:40785/0 (socket says 172.21.15.104:40785) 2024-03-29T11:19:56.812 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.835+0000 7fb72e3d0640 1 -- 172.21.15.104:0/969808678 learned_addr learned my addr 172.21.15.104:0/969808678 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:19:56.812 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.835+0000 7fb72fe33640 10 monclient(hunting): authenticate will time out at 978.086590s 2024-03-29T11:19:56.813 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.836+0000 7fb72cbcd640 1 -- 172.21.15.104:0/969808678 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2232324706 0 0) 0x7fb728145eb0 con 0x7fb7281457f0 2024-03-29T11:19:56.813 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.836+0000 7fb72cbcd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:56.813 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.836+0000 7fb72cbcd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:56.813 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.836+0000 7fb72cbcd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:56.813 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.836+0000 7fb72cbcd640 10 monclient(hunting): my global_id is 4253 2024-03-29T11:19:56.813 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.836+0000 7fb72cbcd640 1 -- 172.21.15.104:0/969808678 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb714003520 con 0x7fb7281457f0 2024-03-29T11:19:56.813 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.836+0000 7fb72cbcd640 1 -- 172.21.15.104:0/969808678 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 765769059 0 0) 0x7fb728145bd0 con 0x7fb72813f650 2024-03-29T11:19:56.813 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.836+0000 7fb72cbcd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:56.813 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.836+0000 7fb72cbcd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:56.813 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.836+0000 7fb72cbcd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:56.813 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.836+0000 7fb72cbcd640 10 monclient(hunting): my global_id is 4240 2024-03-29T11:19:56.813 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.836+0000 7fb72cbcd640 1 -- 172.21.15.104:0/969808678 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb728145eb0 con 0x7fb72813f650 2024-03-29T11:19:56.814 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.837+0000 7fb72cbcd640 1 -- 172.21.15.104:0/969808678 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 565272105 0 0) 0x7fb714003520 con 0x7fb7281457f0 2024-03-29T11:19:56.814 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.837+0000 7fb72cbcd640 1 -- 172.21.15.104:0/969808678 >> v1:172.21.15.119:6789/0 conn(0x7fb72813f650 legacy=0x7fb7280b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:56.814 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.837+0000 7fb72cbcd640 1 -- 172.21.15.104:0/969808678 >> v1:172.21.15.104:6789/0 conn(0x7fb7280a9d70 legacy=0x7fb72814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:56.814 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.837+0000 7fb72cbcd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:56.814 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.837+0000 7fb72cbcd640 1 monclient: found mon.noname-b 2024-03-29T11:19:56.814 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.837+0000 7fb72cbcd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:56.814 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.837+0000 7fb72cbcd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.104:6790/0 2024-03-29T11:19:56.815 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.837+0000 7fb72cbcd640 1 -- 172.21.15.104:0/969808678 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb728146340 con 0x7fb7281457f0 2024-03-29T11:19:56.815 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.837+0000 7fb72cbcd640 10 monclient: _finish_auth 0 2024-03-29T11:19:56.815 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.837+0000 7fb72cbcd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:56.815 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.837+0000 7fb72cbcd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:56.815 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.837+0000 7fb72cbcd640 1 -- 172.21.15.104:0/969808678 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fb7240015c0 con 0x7fb7281457f0 2024-03-29T11:19:56.815 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.837+0000 7fb72cbcd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:56.815 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.837+0000 7fb72cbcd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:56.815 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.837+0000 7fb72cbcd640 10 monclient: dump: 2024-03-29T11:19:56.815 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:19:56.815 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:56.815 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:56.815 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:56.815 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:56.815 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:19:56.816 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:56.816 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:56.816 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:56.816 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:19:56.816 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.837+0000 7fb72cbcd640 1 -- 172.21.15.104:0/969808678 <== mon.2 v1:172.21.15.104:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb724001720 con 0x7fb7281457f0 2024-03-29T11:19:56.816 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.837+0000 7fb72cbcd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:56.816 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.837+0000 7fb72cbcd640 1 -- 172.21.15.104:0/969808678 <== mon.2 v1:172.21.15.104:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fb724001b70 con 0x7fb7281457f0 2024-03-29T11:19:56.816 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.837+0000 7fb72cbcd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:56.816 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.837+0000 7fb72cbcd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:56.816 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.837+0000 7fb72cbcd640 10 monclient: dump: 2024-03-29T11:19:56.816 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:19:56.816 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:56.816 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:56.816 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:56.816 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:56.817 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:19:56.817 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:56.817 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:56.817 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:56.817 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:19:56.818 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.838+0000 7fb72fe33640 5 monclient: authenticate success, global_id 4253 2024-03-29T11:19:56.818 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.838+0000 7fb72fe33640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:56.818 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.838+0000 7fb72fe33640 4 set_mon_vals no callback set 2024-03-29T11:19:56.818 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.839+0000 7fb72fe33640 10 monclient: shutdown 2024-03-29T11:19:56.818 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.839+0000 7fb72fe33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:56.818 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.839+0000 7fb72fe33640 1 -- 172.21.15.104:0/969808678 >> v1:172.21.15.104:6790/0 conn(0x7fb7281457f0 legacy=0x7fb72813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:56.818 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.839+0000 7fb72fe33640 1 -- 172.21.15.104:0/969808678 shutdown_connections 2024-03-29T11:19:56.818 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.839+0000 7fb72fe33640 1 -- 172.21.15.104:0/969808678 >> 172.21.15.104:0/969808678 conn(0x7fb728066a00 msgr2=0x7fb728066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:56.818 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.842+0000 7fb72fe33640 1 -- 172.21.15.104:0/969808678 shutdown_connections 2024-03-29T11:19:56.821 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.842+0000 7fb72fe33640 1 -- 172.21.15.104:0/969808678 wait complete. 2024-03-29T11:19:56.821 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.843+0000 7fb72fe33640 10 monclient: build_initial_monmap 2024-03-29T11:19:56.821 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.843+0000 7fb72fe33640 10 monclient: monmap: 2024-03-29T11:19:56.821 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:19:56.821 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:56.821 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:19:56.822 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:19:56.822 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:56.822 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:19:56.822 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:19:56.822 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:19:56.822 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:19:56.822 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:19:56.822 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.844+0000 7fb72fe33640 1 librados: starting msgr at 2024-03-29T11:19:56.822 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.844+0000 7fb72fe33640 1 librados: starting objecter 2024-03-29T11:19:56.822 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.844+0000 7fb72fe33640 1 Processor -- start 2024-03-29T11:19:56.822 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.844+0000 7fb72fe33640 1 -- start start 2024-03-29T11:19:56.822 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.844+0000 7fb72fe33640 1 librados: setting wanted keys 2024-03-29T11:19:56.822 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.844+0000 7fb72fe33640 1 librados: calling monclient init 2024-03-29T11:19:56.822 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.844+0000 7fb72fe33640 10 monclient: init 2024-03-29T11:19:56.823 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.846+0000 7fb72fe33640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:56.823 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.846+0000 7fb72fe33640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:56.823 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.846+0000 7fb72fe33640 10 monclient(hunting): picked mon.noname-a con 0x7fb7280a9d70 addr v1:172.21.15.104:6789/0 2024-03-29T11:19:56.823 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.846+0000 7fb72fe33640 10 monclient(hunting): picked mon.noname-b con 0x7fb72813f650 addr v1:172.21.15.119:6789/0 2024-03-29T11:19:56.823 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.846+0000 7fb72fe33640 10 monclient(hunting): picked mon.noname-c con 0x7fb7281457f0 addr v1:172.21.15.104:6790/0 2024-03-29T11:19:56.823 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.846+0000 7fb72fe33640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb728142000 con 0x7fb7280a9d70 2024-03-29T11:19:56.823 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.846+0000 7fb72fe33640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7281422e0 con 0x7fb72813f650 2024-03-29T11:19:56.823 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.846+0000 7fb72fe33640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb728140180 con 0x7fb7281457f0 2024-03-29T11:19:56.823 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.846+0000 7fb72fe33640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:56.823 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.846+0000 7fb72fe33640 10 monclient(hunting): authenticate will time out at 978.097313s 2024-03-29T11:19:56.826 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.849+0000 7fb72d3ce640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fb72813f650 0x7fb7281350d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:54584/0 (socket says 172.21.15.104:54584) 2024-03-29T11:19:56.826 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.849+0000 7fb72dbcf640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fb7280a9d70 0x7fb7281406a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:40923/0 (socket says 172.21.15.104:40923) 2024-03-29T11:19:56.826 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.849+0000 7fb72dbcf640 1 -- 172.21.15.104:0/1665847002 learned_addr learned my addr 172.21.15.104:0/1665847002 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:19:56.831 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.850+0000 7fb71a7fc640 10 client.?.objecter ms_handle_connect 0x7fb7280a9d70 2024-03-29T11:19:56.831 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.850+0000 7fb71a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:56.831 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.850+0000 7fb71a7fc640 10 client.?.objecter ms_handle_connect 0x7fb72813f650 2024-03-29T11:19:56.832 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.850+0000 7fb71a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:56.832 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.850+0000 7fb71a7fc640 1 -- 172.21.15.104:0/1665847002 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3905225227 0 0) 0x7fb728142000 con 0x7fb7280a9d70 2024-03-29T11:19:56.832 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.850+0000 7fb71a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:56.832 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.850+0000 7fb71a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:56.832 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.850+0000 7fb71a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:56.832 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.850+0000 7fb71a7fc640 10 monclient(hunting): my global_id is 4254 2024-03-29T11:19:56.832 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.850+0000 7fb71a7fc640 1 -- 172.21.15.104:0/1665847002 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb70c004eb0 con 0x7fb7280a9d70 2024-03-29T11:19:56.832 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.851+0000 7fb71a7fc640 1 -- 172.21.15.104:0/1665847002 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1800013741 0 0) 0x7fb7281422e0 con 0x7fb72813f650 2024-03-29T11:19:56.832 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.851+0000 7fb71a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:56.832 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.851+0000 7fb71a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:56.832 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.851+0000 7fb71a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:56.832 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.851+0000 7fb71a7fc640 10 monclient(hunting): my global_id is 4243 2024-03-29T11:19:56.832 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.851+0000 7fb71a7fc640 1 -- 172.21.15.104:0/1665847002 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb728142000 con 0x7fb72813f650 2024-03-29T11:19:56.833 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.851+0000 7fb71a7fc640 1 -- 172.21.15.104:0/1665847002 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3113379318 0 0) 0x7fb70c004eb0 con 0x7fb7280a9d70 2024-03-29T11:19:56.833 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.851+0000 7fb71a7fc640 1 -- 172.21.15.104:0/1665847002 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7281422e0 con 0x7fb7280a9d70 2024-03-29T11:19:56.833 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.851+0000 7fb71a7fc640 10 client.?.objecter ms_handle_connect 0x7fb7281457f0 2024-03-29T11:19:56.833 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.851+0000 7fb71a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:56.833 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.851+0000 7fb71a7fc640 1 -- 172.21.15.104:0/1665847002 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fb724001440 con 0x7fb7280a9d70 2024-03-29T11:19:56.833 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.851+0000 7fb71a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:56.833 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.851+0000 7fb71a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:56.833 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.851+0000 7fb71a7fc640 10 monclient(hunting): dump: 2024-03-29T11:19:56.833 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:19:56.833 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:56.833 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:56.833 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:56.833 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:56.833 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:19:56.833 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:56.834 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:56.834 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:56.834 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:19:56.834 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.851+0000 7fb71a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:56.834 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.852+0000 7fb71a7fc640 1 -- 172.21.15.104:0/1665847002 <== mon.0 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1361618889 0 0) 0x7fb7281422e0 con 0x7fb7280a9d70 2024-03-29T11:19:56.834 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.852+0000 7fb71a7fc640 1 -- 172.21.15.104:0/1665847002 >> v1:172.21.15.104:6790/0 conn(0x7fb7281457f0 legacy=0x7fb7281e2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:56.834 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.852+0000 7fb71a7fc640 1 -- 172.21.15.104:0/1665847002 >> v1:172.21.15.119:6789/0 conn(0x7fb72813f650 legacy=0x7fb7281350d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:56.834 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.852+0000 7fb71a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:19:56.834 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.852+0000 7fb71a7fc640 1 monclient: found mon.a 2024-03-29T11:19:56.834 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.852+0000 7fb71a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:56.834 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.852+0000 7fb71a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:19:56.834 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.852+0000 7fb71a7fc640 1 -- 172.21.15.104:0/1665847002 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb728140db0 con 0x7fb7280a9d70 2024-03-29T11:19:56.834 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.852+0000 7fb71a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:19:56.834 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.852+0000 7fb71a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:19:56.834 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.852+0000 7fb71a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:56.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.852+0000 7fb72fe33640 5 monclient: authenticate success, global_id 4254 2024-03-29T11:19:56.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.852+0000 7fb72fe33640 10 monclient: _renew_subs 2024-03-29T11:19:56.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.852+0000 7fb72fe33640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:19:56.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.852+0000 7fb72fe33640 1 -- 172.21.15.104:0/1665847002 --> v1:172.21.15.104:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7281410c0 con 0x7fb7280a9d70 2024-03-29T11:19:56.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.852+0000 7fb72fe33640 10 client.4254.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:56.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.852+0000 7fb72fe33640 10 monclient: _renew_subs 2024-03-29T11:19:56.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.852+0000 7fb72fe33640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:19:56.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.852+0000 7fb72fe33640 1 -- 172.21.15.104:0/1665847002 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb728139110 con 0x7fb7280a9d70 2024-03-29T11:19:56.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.852+0000 7fb72fe33640 1 librados: init done 2024-03-29T11:19:56.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.853+0000 7fb71a7fc640 1 -- 172.21.15.104:0/1665847002 <== mon.0 v1:172.21.15.104:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb7240015a0 con 0x7fb7280a9d70 2024-03-29T11:19:56.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.853+0000 7fb71a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:56.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.853+0000 7fb71a7fc640 1 -- 172.21.15.104:0/1665847002 <== mon.0 v1:172.21.15.104:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fb724003b40 con 0x7fb7280a9d70 2024-03-29T11:19:56.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.853+0000 7fb71a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:56.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.853+0000 7fb71a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:56.836 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.853+0000 7fb71bfff640 4 set_mon_vals no callback set 2024-03-29T11:19:56.836 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.853+0000 7fb71a7fc640 10 monclient: dump: 2024-03-29T11:19:56.836 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:19:56.836 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:19:56.836 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:56.836 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:19:56.836 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:56.836 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:19:56.836 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:19:56.836 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:19:56.836 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:19:56.836 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:19:56.838 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.854+0000 7fb72fe33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:56.838 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.854+0000 7fb72fe33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:56.838 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.854+0000 7fb72fe33640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:19:56.838 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.854+0000 7fb72fe33640 1 -- 172.21.15.104:0/1665847002 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb728140310 con 0x7fb7280a9d70 2024-03-29T11:19:56.838 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.855+0000 7fb71a7fc640 1 -- 172.21.15.104:0/1665847002 <== mon.0 v1:172.21.15.104:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7fb724003ca0 con 0x7fb7280a9d70 2024-03-29T11:19:56.838 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.856+0000 7fb72d3ce640 10 monclient: get_auth_request con 0x7fb70c06dd90 auth_method 0 2024-03-29T11:19:56.838 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.857+0000 7fb71a7fc640 10 client.4254.objecter ms_handle_connect 0x7fb70c06dd90 2024-03-29T11:19:56.838 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.857+0000 7fb71a7fc640 1 -- 172.21.15.104:0/1665847002 <== mon.0 v1:172.21.15.104:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7fb724084b40 con 0x7fb7280a9d70 2024-03-29T11:19:56.838 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.857+0000 7fb71a7fc640 10 client.4254.objecter ms_dispatch 0x7fb728000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:56.838 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.857+0000 7fb71a7fc640 3 client.4254.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:56.838 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.857+0000 7fb71a7fc640 3 client.4254.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:56.839 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.857+0000 7fb71a7fc640 20 client.4254.objecter dump_active .. 0 homeless 2024-03-29T11:19:56.840 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.863+0000 7fb71a7fc640 1 -- 172.21.15.104:0/1665847002 <== mon.0 v1:172.21.15.104:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb724050420 con 0x7fb7280a9d70 2024-03-29T11:19:56.840 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.863+0000 7fb71a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:56.840 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:56.863+0000 7fb71a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:57.169 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:57.192+0000 7fb72fe33640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:57.169 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:57.192+0000 7fb72fe33640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:57.169 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:57.192+0000 7fb72fe33640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:19:57.169 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:57.192+0000 7fb72fe33640 1 -- 172.21.15.104:0/1665847002 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb72813ff80 con 0x7fb7280a9d70 2024-03-29T11:19:57.170 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:57.194+0000 7fb71a7fc640 1 -- 172.21.15.104:0/1665847002 <== mon.0 v1:172.21.15.104:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7fb724052350 con 0x7fb7280a9d70 2024-03-29T11:19:57.171 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:57.194+0000 7fb71a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:57.171 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:57.194+0000 7fb71a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:57.171 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:19:57.171 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:57.174 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:57.197+0000 7fb72fe33640 10 librados: watch_flush enter 2024-03-29T11:19:57.174 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:57.197+0000 7fb72fe33640 10 librados: watch_flush exit 2024-03-29T11:19:57.174 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:57.197+0000 7fb72fe33640 20 client.4254.objecter shutdown clearing up homeless session... 2024-03-29T11:19:57.174 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:57.197+0000 7fb72fe33640 10 client.4254.objecter successfully canceled tick 2024-03-29T11:19:57.174 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:57.197+0000 7fb72fe33640 1 -- 172.21.15.104:0/1665847002 >> v1:172.21.15.104:6816/3341126660 conn(0x7fb70c06dd90 legacy=0x7fb70c070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.174 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:57.197+0000 7fb72fe33640 10 monclient: shutdown 2024-03-29T11:19:57.174 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:57.197+0000 7fb72fe33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:57.174 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:57.197+0000 7fb72fe33640 1 -- 172.21.15.104:0/1665847002 >> v1:172.21.15.104:6789/0 conn(0x7fb7280a9d70 legacy=0x7fb7281406a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:57.174 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:57.197+0000 7fb72fe33640 1 -- 172.21.15.104:0/1665847002 shutdown_connections 2024-03-29T11:19:57.174 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:57.197+0000 7fb72fe33640 1 -- 172.21.15.104:0/1665847002 >> 172.21.15.104:0/1665847002 conn(0x7fb728066a00 msgr2=0x7fb7280b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:57.175 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:57.199+0000 7fb72fe33640 1 -- 172.21.15.104:0/1665847002 shutdown_connections 2024-03-29T11:19:57.176 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:57.199+0000 7fb72fe33640 1 -- 172.21.15.104:0/1665847002 wait complete. 2024-03-29T11:19:57.176 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:19:57.199+0000 7fb72fe33640 1 librados: shutdown 2024-03-29T11:19:57.192 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:03.999 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:04.148 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.171+0000 7fbf433dc640 10 monclient: get_monmap_and_config 2024-03-29T11:20:04.148 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.171+0000 7fbf433dc640 10 monclient: build_initial_monmap 2024-03-29T11:20:04.148 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.171+0000 7fbf433dc640 10 monclient: monmap: 2024-03-29T11:20:04.148 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:20:04.148 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:20:04.148 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:20:04.172291+0000 2024-03-29T11:20:04.148 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:20:04.172291+0000 2024-03-29T11:20:04.148 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:04.148 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:20:04.149 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:20:04.149 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:20:04.149 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:20:04.149 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:20:04.149 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.172+0000 7fbf433dc640 1 Processor -- start 2024-03-29T11:20:04.149 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.172+0000 7fbf433dc640 1 -- start start 2024-03-29T11:20:04.149 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.172+0000 7fbf433dc640 10 monclient: init 2024-03-29T11:20:04.150 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.173+0000 7fbf433dc640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:04.150 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.173+0000 7fbf433dc640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:20:04.153 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.177+0000 7fbf423da640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7fbf340389e0 0x7fbf34033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:59677/0 (socket says 172.21.15.104:59677) 2024-03-29T11:20:04.153 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.177+0000 7fbf423da640 1 -- 172.21.15.104:0/1477398274 learned_addr learned my addr 172.21.15.104:0/1477398274 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:20:04.154 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.177+0000 7fbf433dc640 10 monclient(hunting): picked mon.noname-b con 0x7fbf340389e0 addr v1:172.21.15.104:6790/0 2024-03-29T11:20:04.154 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.178+0000 7fbf433dc640 10 monclient(hunting): picked mon.noname-c con 0x7fbf34038dc0 addr v1:172.21.15.119:6789/0 2024-03-29T11:20:04.156 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.179+0000 7fbf433dc640 10 monclient(hunting): picked mon.noname-a con 0x7fbf34030660 addr v1:172.21.15.104:6789/0 2024-03-29T11:20:04.156 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.179+0000 7fbf433dc640 1 -- 172.21.15.104:0/1477398274 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf340c54c0 con 0x7fbf34030660 2024-03-29T11:20:04.156 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.179+0000 7fbf433dc640 1 -- 172.21.15.104:0/1477398274 --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf34036430 con 0x7fbf34038dc0 2024-03-29T11:20:04.156 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.179+0000 7fbf433dc640 1 -- 172.21.15.104:0/1477398274 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf34039840 con 0x7fbf340389e0 2024-03-29T11:20:04.156 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.179+0000 7fbf433dc640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:04.157 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.179+0000 7fbf433dc640 10 monclient(hunting): authenticate will time out at 985.430566s 2024-03-29T11:20:04.157 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.179+0000 7fbf413d8640 1 -- 172.21.15.104:0/1477398274 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3846565452 0 0) 0x7fbf34039840 con 0x7fbf340389e0 2024-03-29T11:20:04.157 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.179+0000 7fbf413d8640 10 monclient(hunting): _negotiate 2024-03-29T11:20:04.157 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.179+0000 7fbf413d8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:04.157 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.179+0000 7fbf413d8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:04.157 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.179+0000 7fbf413d8640 10 monclient(hunting): my global_id is 4259 2024-03-29T11:20:04.157 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.179+0000 7fbf413d8640 1 -- 172.21.15.104:0/1477398274 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf28003520 con 0x7fbf340389e0 2024-03-29T11:20:04.157 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.180+0000 7fbf413d8640 1 -- 172.21.15.104:0/1477398274 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 327226834 0 0) 0x7fbf28003520 con 0x7fbf340389e0 2024-03-29T11:20:04.157 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.180+0000 7fbf413d8640 1 -- 172.21.15.104:0/1477398274 >> v1:172.21.15.119:6789/0 conn(0x7fbf34038dc0 legacy=0x7fbf340cadd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:20:04.160 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.181+0000 7fbf413d8640 1 -- 172.21.15.104:0/1477398274 >> v1:172.21.15.104:6789/0 conn(0x7fbf34030660 legacy=0x7fbf340d3170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:20:04.160 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.181+0000 7fbf413d8640 10 monclient: _finish_hunting 0 2024-03-29T11:20:04.160 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.181+0000 7fbf413d8640 1 monclient: found mon.noname-b 2024-03-29T11:20:04.160 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.181+0000 7fbf413d8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:04.160 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.181+0000 7fbf413d8640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.104:6790/0 2024-03-29T11:20:04.160 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.181+0000 7fbf413d8640 1 -- 172.21.15.104:0/1477398274 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf34035690 con 0x7fbf340389e0 2024-03-29T11:20:04.160 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.181+0000 7fbf413d8640 10 monclient: _finish_auth 0 2024-03-29T11:20:04.160 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.181+0000 7fbf413d8640 10 monclient: _check_auth_tickets 2024-03-29T11:20:04.160 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.181+0000 7fbf413d8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:04.160 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.181+0000 7fbf413d8640 1 -- 172.21.15.104:0/1477398274 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fbf38001690 con 0x7fbf340389e0 2024-03-29T11:20:04.160 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.181+0000 7fbf413d8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:04.160 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.181+0000 7fbf413d8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:04.160 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.181+0000 7fbf413d8640 10 monclient: dump: 2024-03-29T11:20:04.161 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:20:04.161 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:20:04.161 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:04.161 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:04.161 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:04.161 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:20:04.161 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:20:04.161 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:20:04.161 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:20:04.161 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:20:04.161 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.183+0000 7fbf433dc640 5 monclient: authenticate success, global_id 4259 2024-03-29T11:20:04.161 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.183+0000 7fbf433dc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:04.161 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.183+0000 7fbf413d8640 1 -- 172.21.15.104:0/1477398274 <== mon.2 v1:172.21.15.104:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf380018f0 con 0x7fbf340389e0 2024-03-29T11:20:04.161 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.183+0000 7fbf413d8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:04.161 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.183+0000 7fbf413d8640 1 -- 172.21.15.104:0/1477398274 <== mon.2 v1:172.21.15.104:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fbf38001d00 con 0x7fbf340389e0 2024-03-29T11:20:04.162 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.183+0000 7fbf413d8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:04.162 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.183+0000 7fbf413d8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:04.162 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.183+0000 7fbf413d8640 10 monclient: dump: 2024-03-29T11:20:04.162 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:20:04.162 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:20:04.162 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:04.162 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:04.162 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:04.162 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:20:04.162 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:20:04.162 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:20:04.162 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:20:04.162 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:20:04.162 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.183+0000 7fbf433dc640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:04.162 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.183+0000 7fbf433dc640 4 set_mon_vals no callback set 2024-03-29T11:20:04.163 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.184+0000 7fbf433dc640 10 monclient: shutdown 2024-03-29T11:20:04.164 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.184+0000 7fbf433dc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:04.164 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.184+0000 7fbf433dc640 1 -- 172.21.15.104:0/1477398274 >> v1:172.21.15.104:6790/0 conn(0x7fbf340389e0 legacy=0x7fbf34033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:04.164 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.185+0000 7fbf433dc640 1 -- 172.21.15.104:0/1477398274 shutdown_connections 2024-03-29T11:20:04.164 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.185+0000 7fbf433dc640 1 -- 172.21.15.104:0/1477398274 >> 172.21.15.104:0/1477398274 conn(0x7fbf34005c00 msgr2=0x7fbf34006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:04.164 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.185+0000 7fbf433dc640 1 -- 172.21.15.104:0/1477398274 shutdown_connections 2024-03-29T11:20:04.164 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.185+0000 7fbf433dc640 1 -- 172.21.15.104:0/1477398274 wait complete. 2024-03-29T11:20:04.164 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.187+0000 7fbf433dc640 10 monclient: build_initial_monmap 2024-03-29T11:20:04.164 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.187+0000 7fbf433dc640 10 monclient: monmap: 2024-03-29T11:20:04.164 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:20:04.164 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:04.164 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:20:04.164 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:20:04.164 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:04.164 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:20:04.165 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:20:04.165 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:20:04.165 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:20:04.165 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:20:04.167 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.188+0000 7fbf433dc640 1 librados: starting msgr at 2024-03-29T11:20:04.167 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.188+0000 7fbf433dc640 1 librados: starting objecter 2024-03-29T11:20:04.167 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.188+0000 7fbf433dc640 1 Processor -- start 2024-03-29T11:20:04.167 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.188+0000 7fbf433dc640 1 -- start start 2024-03-29T11:20:04.167 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.188+0000 7fbf433dc640 1 librados: setting wanted keys 2024-03-29T11:20:04.167 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.188+0000 7fbf433dc640 1 librados: calling monclient init 2024-03-29T11:20:04.167 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.188+0000 7fbf433dc640 10 monclient: init 2024-03-29T11:20:04.167 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.189+0000 7fbf433dc640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:04.167 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.189+0000 7fbf433dc640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:20:04.167 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.189+0000 7fbf433dc640 10 monclient(hunting): picked mon.noname-b con 0x7fbf34030660 addr v1:172.21.15.119:6789/0 2024-03-29T11:20:04.167 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.189+0000 7fbf433dc640 10 monclient(hunting): picked mon.noname-c con 0x7fbf340389e0 addr v1:172.21.15.104:6790/0 2024-03-29T11:20:04.167 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.189+0000 7fbf433dc640 10 monclient(hunting): picked mon.noname-a con 0x7fbf3402eda0 addr v1:172.21.15.104:6789/0 2024-03-29T11:20:04.168 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.189+0000 7fbf433dc640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf34009b60 con 0x7fbf3402eda0 2024-03-29T11:20:04.168 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.189+0000 7fbf433dc640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf3416d5d0 con 0x7fbf34030660 2024-03-29T11:20:04.168 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.189+0000 7fbf433dc640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf3416d280 con 0x7fbf340389e0 2024-03-29T11:20:04.168 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.189+0000 7fbf433dc640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:04.168 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.189+0000 7fbf433dc640 10 monclient(hunting): authenticate will time out at 985.440435s 2024-03-29T11:20:04.169 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.192+0000 7fbf42bdb640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fbf3402eda0 0x7fbf34174140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:35847/0 (socket says 172.21.15.104:35847) 2024-03-29T11:20:04.170 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.192+0000 7fbf42bdb640 1 -- 172.21.15.104:0/573628045 learned_addr learned my addr 172.21.15.104:0/573628045 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:20:04.170 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.192+0000 7fbf32ffd640 10 client.?.objecter ms_handle_connect 0x7fbf3402eda0 2024-03-29T11:20:04.170 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.193+0000 7fbf32ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:04.170 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.193+0000 7fbf32ffd640 1 -- 172.21.15.104:0/573628045 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 648516266 0 0) 0x7fbf34009b60 con 0x7fbf3402eda0 2024-03-29T11:20:04.170 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.193+0000 7fbf32ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:04.170 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.193+0000 7fbf32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:04.170 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.193+0000 7fbf32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:04.170 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.193+0000 7fbf32ffd640 10 monclient(hunting): my global_id is 4260 2024-03-29T11:20:04.170 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.193+0000 7fbf32ffd640 1 -- 172.21.15.104:0/573628045 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf20004eb0 con 0x7fbf3402eda0 2024-03-29T11:20:04.171 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.194+0000 7fbf32ffd640 1 -- 172.21.15.104:0/573628045 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3343684271 0 0) 0x7fbf20004eb0 con 0x7fbf3402eda0 2024-03-29T11:20:04.171 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.194+0000 7fbf32ffd640 1 -- 172.21.15.104:0/573628045 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf34009b60 con 0x7fbf3402eda0 2024-03-29T11:20:04.171 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.194+0000 7fbf32ffd640 1 -- 172.21.15.104:0/573628045 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fbf3c04fdd0 con 0x7fbf3402eda0 2024-03-29T11:20:04.171 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.194+0000 7fbf32ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:04.171 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.194+0000 7fbf32ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:04.172 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.194+0000 7fbf32ffd640 10 monclient(hunting): dump: 2024-03-29T11:20:04.172 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:20:04.172 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:20:04.172 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:04.172 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:04.172 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:04.172 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:20:04.172 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:20:04.172 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:20:04.172 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:20:04.172 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:20:04.172 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.194+0000 7fbf32ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:04.172 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.194+0000 7fbf32ffd640 1 -- 172.21.15.104:0/573628045 <== mon.0 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2275815877 0 0) 0x7fbf34009b60 con 0x7fbf3402eda0 2024-03-29T11:20:04.172 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.195+0000 7fbf32ffd640 1 -- 172.21.15.104:0/573628045 >> v1:172.21.15.104:6790/0 conn(0x7fbf340389e0 legacy=0x7fbf34169670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:20:04.173 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.197+0000 7fbf32ffd640 1 -- 172.21.15.104:0/573628045 >> v1:172.21.15.119:6789/0 conn(0x7fbf34030660 legacy=0x7fbf3416cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:04.175 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.197+0000 7fbf32ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:04.175 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.197+0000 7fbf32ffd640 1 monclient: found mon.a 2024-03-29T11:20:04.176 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.198+0000 7fbf32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:04.176 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.198+0000 7fbf32ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:20:04.176 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.198+0000 7fbf32ffd640 1 -- 172.21.15.104:0/573628045 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf34169d80 con 0x7fbf3402eda0 2024-03-29T11:20:04.176 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.198+0000 7fbf32ffd640 10 monclient: _finish_auth 0 2024-03-29T11:20:04.176 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.198+0000 7fbf32ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:04.176 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.198+0000 7fbf32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:04.176 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.198+0000 7fbf32ffd640 10 client.?.objecter ms_handle_connect 0x7fbf34030660 2024-03-29T11:20:04.176 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.198+0000 7fbf32ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:04.176 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.198+0000 7fbf433dc640 5 monclient: authenticate success, global_id 4260 2024-03-29T11:20:04.176 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.198+0000 7fbf433dc640 10 monclient: _renew_subs 2024-03-29T11:20:04.176 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.198+0000 7fbf433dc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:20:04.176 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.198+0000 7fbf433dc640 1 -- 172.21.15.104:0/573628045 --> v1:172.21.15.104:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf34169fd0 con 0x7fbf3402eda0 2024-03-29T11:20:04.176 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.198+0000 7fbf433dc640 10 client.4260.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:04.176 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.198+0000 7fbf433dc640 10 monclient: _renew_subs 2024-03-29T11:20:04.176 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.198+0000 7fbf433dc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:20:04.176 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.198+0000 7fbf433dc640 1 -- 172.21.15.104:0/573628045 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf3416a280 con 0x7fbf3402eda0 2024-03-29T11:20:04.177 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.198+0000 7fbf433dc640 1 librados: init done 2024-03-29T11:20:04.177 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.200+0000 7fbf433dc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:04.177 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.200+0000 7fbf433dc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:04.177 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.200+0000 7fbf433dc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:20:04.177 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.200+0000 7fbf433dc640 1 -- 172.21.15.104:0/573628045 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf34007b90 con 0x7fbf3402eda0 2024-03-29T11:20:04.178 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.201+0000 7fbf32ffd640 1 -- 172.21.15.104:0/573628045 <== mon.0 v1:172.21.15.104:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf3c04f400 con 0x7fbf3402eda0 2024-03-29T11:20:04.178 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.201+0000 7fbf32ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:04.178 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.202+0000 7fbf33fff640 4 set_mon_vals no callback set 2024-03-29T11:20:04.179 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.202+0000 7fbf32ffd640 1 -- 172.21.15.104:0/573628045 <== mon.0 v1:172.21.15.104:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fbf3c075f10 con 0x7fbf3402eda0 2024-03-29T11:20:04.179 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.202+0000 7fbf32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:04.179 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.202+0000 7fbf32ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:04.179 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.202+0000 7fbf32ffd640 10 monclient: dump: 2024-03-29T11:20:04.179 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:20:04.179 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:20:04.179 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:04.179 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:04.179 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:04.179 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:20:04.179 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:20:04.179 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:20:04.179 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:20:04.180 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:20:04.180 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.204+0000 7fbf32ffd640 1 -- 172.21.15.104:0/573628045 <== mon.0 v1:172.21.15.104:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7fbf3c067bf0 con 0x7fbf3402eda0 2024-03-29T11:20:04.183 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.207+0000 7fbf32ffd640 1 -- 172.21.15.104:0/573628045 <== mon.0 v1:172.21.15.104:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7fbf3c0ff600 con 0x7fbf3402eda0 2024-03-29T11:20:04.183 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.207+0000 7fbf32ffd640 10 client.4260.objecter ms_dispatch 0x7fbf3c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:04.184 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.207+0000 7fbf32ffd640 3 client.4260.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:04.184 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.207+0000 7fbf32ffd640 3 client.4260.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:04.184 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.207+0000 7fbf32ffd640 20 client.4260.objecter dump_active .. 0 homeless 2024-03-29T11:20:04.185 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.208+0000 7fbf423da640 10 monclient: get_auth_request con 0x7fbf2006dab0 auth_method 0 2024-03-29T11:20:04.185 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.209+0000 7fbf32ffd640 10 client.4260.objecter ms_handle_connect 0x7fbf2006dab0 2024-03-29T11:20:04.189 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.212+0000 7fbf32ffd640 1 -- 172.21.15.104:0/573628045 <== mon.0 v1:172.21.15.104:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbf3c0caf40 con 0x7fbf3402eda0 2024-03-29T11:20:04.189 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.213+0000 7fbf32ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:04.189 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.213+0000 7fbf32ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:04.521 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.544+0000 7fbf433dc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:04.521 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.544+0000 7fbf433dc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:04.521 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.544+0000 7fbf433dc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:20:04.521 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.544+0000 7fbf433dc640 1 -- 172.21.15.104:0/573628045 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf3416d410 con 0x7fbf3402eda0 2024-03-29T11:20:04.523 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.546+0000 7fbf32ffd640 1 -- 172.21.15.104:0/573628045 <== mon.0 v1:172.21.15.104:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7fbf3c0cce70 con 0x7fbf3402eda0 2024-03-29T11:20:04.523 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.547+0000 7fbf32ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:04.523 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.547+0000 7fbf32ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:04.524 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:20:04.524 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:04.527 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.550+0000 7fbf433dc640 10 librados: watch_flush enter 2024-03-29T11:20:04.527 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.550+0000 7fbf433dc640 10 librados: watch_flush exit 2024-03-29T11:20:04.527 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.550+0000 7fbf433dc640 20 client.4260.objecter shutdown clearing up homeless session... 2024-03-29T11:20:04.527 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.550+0000 7fbf433dc640 10 client.4260.objecter successfully canceled tick 2024-03-29T11:20:04.529 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.553+0000 7fbf433dc640 1 -- 172.21.15.104:0/573628045 >> v1:172.21.15.104:6816/3341126660 conn(0x7fbf2006dab0 legacy=0x7fbf2006ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:04.529 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.553+0000 7fbf433dc640 10 monclient: shutdown 2024-03-29T11:20:04.529 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.553+0000 7fbf433dc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:04.529 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.553+0000 7fbf433dc640 1 -- 172.21.15.104:0/573628045 >> v1:172.21.15.104:6789/0 conn(0x7fbf3402eda0 legacy=0x7fbf34174140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:04.529 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.553+0000 7fbf433dc640 1 -- 172.21.15.104:0/573628045 shutdown_connections 2024-03-29T11:20:04.530 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.553+0000 7fbf433dc640 1 -- 172.21.15.104:0/573628045 >> 172.21.15.104:0/573628045 conn(0x7fbf34005c00 msgr2=0x7fbf340c3710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:04.532 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.556+0000 7fbf433dc640 1 -- 172.21.15.104:0/573628045 shutdown_connections 2024-03-29T11:20:04.532 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.556+0000 7fbf433dc640 1 -- 172.21.15.104:0/573628045 wait complete. 2024-03-29T11:20:04.533 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:04.556+0000 7fbf433dc640 1 librados: shutdown 2024-03-29T11:20:04.551 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:11.358 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:11.513 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.535+0000 7f55dd3c4640 10 monclient: get_monmap_and_config 2024-03-29T11:20:11.513 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.535+0000 7f55dd3c4640 10 monclient: build_initial_monmap 2024-03-29T11:20:11.513 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.535+0000 7f55dd3c4640 10 monclient: monmap: 2024-03-29T11:20:11.513 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:20:11.513 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:20:11.513 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:20:11.535992+0000 2024-03-29T11:20:11.513 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:20:11.535992+0000 2024-03-29T11:20:11.514 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:11.514 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:20:11.514 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:20:11.514 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:20:11.514 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:20:11.514 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:20:11.516 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.536+0000 7f55dd3c4640 1 Processor -- start 2024-03-29T11:20:11.516 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.536+0000 7f55dd3c4640 1 -- start start 2024-03-29T11:20:11.516 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.536+0000 7f55dd3c4640 10 monclient: init 2024-03-29T11:20:11.516 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.538+0000 7f55dd3c4640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:11.516 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.538+0000 7f55dd3c4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:20:11.517 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.538+0000 7f55dd3c4640 10 monclient(hunting): picked mon.noname-b con 0x7f55d80a9930 addr v1:172.21.15.104:6790/0 2024-03-29T11:20:11.517 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.538+0000 7f55dd3c4640 10 monclient(hunting): picked mon.noname-a con 0x7f55d8149dc0 addr v1:172.21.15.104:6789/0 2024-03-29T11:20:11.517 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.538+0000 7f55dd3c4640 10 monclient(hunting): picked mon.noname-c con 0x7f55d814ff40 addr v1:172.21.15.119:6789/0 2024-03-29T11:20:11.517 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.538+0000 7f55dd3c4640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55d80aae60 con 0x7f55d8149dc0 2024-03-29T11:20:11.517 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.538+0000 7f55dd3c4640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55d80b4310 con 0x7f55d814ff40 2024-03-29T11:20:11.517 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.538+0000 7f55dd3c4640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55d80afdd0 con 0x7f55d80a9930 2024-03-29T11:20:11.517 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.538+0000 7f55dd3c4640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:11.517 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.538+0000 7f55dd3c4640 10 monclient(hunting): authenticate will time out at 992.789353s 2024-03-29T11:20:11.521 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.544+0000 7f55d77fe640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f55d814ff40 0x7f55d8152330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:45307/0 (socket says 172.21.15.104:45307) 2024-03-29T11:20:11.521 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.544+0000 7f55d77fe640 1 -- 172.21.15.104:0/3479485160 learned_addr learned my addr 172.21.15.104:0/3479485160 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:20:11.523 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.546+0000 7f55d5ffb640 1 -- 172.21.15.104:0/3479485160 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1890941916 0 0) 0x7f55d80afdd0 con 0x7f55d80a9930 2024-03-29T11:20:11.523 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.546+0000 7f55d5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:20:11.523 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.546+0000 7f55d5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:11.523 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.546+0000 7f55d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:11.523 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.546+0000 7f55d5ffb640 10 monclient(hunting): my global_id is 4262 2024-03-29T11:20:11.523 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.546+0000 7f55d5ffb640 1 -- 172.21.15.104:0/3479485160 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55c0003520 con 0x7f55d80a9930 2024-03-29T11:20:11.523 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.546+0000 7f55d5ffb640 1 -- 172.21.15.104:0/3479485160 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3016510181 0 0) 0x7f55d80b4310 con 0x7f55d814ff40 2024-03-29T11:20:11.523 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.546+0000 7f55d5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:20:11.523 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.546+0000 7f55d5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:11.523 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.546+0000 7f55d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:11.523 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.546+0000 7f55d5ffb640 10 monclient(hunting): my global_id is 4249 2024-03-29T11:20:11.524 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.546+0000 7f55d5ffb640 1 -- 172.21.15.104:0/3479485160 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55c0003840 con 0x7f55d814ff40 2024-03-29T11:20:11.524 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.547+0000 7f55d5ffb640 1 -- 172.21.15.104:0/3479485160 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2591419951 0 0) 0x7f55c0003520 con 0x7f55d80a9930 2024-03-29T11:20:11.524 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.547+0000 7f55d5ffb640 1 -- 172.21.15.104:0/3479485160 >> v1:172.21.15.119:6789/0 conn(0x7f55d814ff40 legacy=0x7f55d8152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:11.524 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.547+0000 7f55d5ffb640 1 -- 172.21.15.104:0/3479485160 >> v1:172.21.15.104:6789/0 conn(0x7f55d8149dc0 legacy=0x7f55d814a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:20:11.524 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.547+0000 7f55d5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:20:11.525 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.547+0000 7f55d5ffb640 1 monclient: found mon.noname-b 2024-03-29T11:20:11.525 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.547+0000 7f55d5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:11.525 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.547+0000 7f55d5ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.104:6790/0 2024-03-29T11:20:11.525 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.547+0000 7f55d5ffb640 1 -- 172.21.15.104:0/3479485160 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55d814a950 con 0x7f55d80a9930 2024-03-29T11:20:11.525 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.547+0000 7f55d5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:20:11.525 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.547+0000 7f55d5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:20:11.525 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.547+0000 7f55d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:11.525 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.547+0000 7f55d5ffb640 1 -- 172.21.15.104:0/3479485160 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f55cc001670 con 0x7f55d80a9930 2024-03-29T11:20:11.525 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.547+0000 7f55d5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:11.525 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.547+0000 7f55d5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:11.525 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.547+0000 7f55d5ffb640 10 monclient: dump: 2024-03-29T11:20:11.525 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:20:11.526 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:20:11.526 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:11.526 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:11.526 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:11.526 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:20:11.526 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:20:11.526 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:20:11.526 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:20:11.526 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:20:11.526 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.547+0000 7f55dd3c4640 5 monclient: authenticate success, global_id 4262 2024-03-29T11:20:11.526 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.547+0000 7f55dd3c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:11.527 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.547+0000 7f55d5ffb640 1 -- 172.21.15.104:0/3479485160 <== mon.2 v1:172.21.15.104:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f55cc001850 con 0x7f55d80a9930 2024-03-29T11:20:11.527 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.547+0000 7f55d5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:11.527 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.547+0000 7f55dd3c4640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:11.527 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.547+0000 7f55dd3c4640 4 set_mon_vals no callback set 2024-03-29T11:20:11.527 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.548+0000 7f55dd3c4640 10 monclient: shutdown 2024-03-29T11:20:11.527 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.548+0000 7f55dd3c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:11.527 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.548+0000 7f55dd3c4640 1 -- 172.21.15.104:0/3479485160 >> v1:172.21.15.104:6790/0 conn(0x7f55d80a9930 legacy=0x7f55d8143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:11.527 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.548+0000 7f55d5ffb640 1 -- 172.21.15.104:0/3479485160 <== mon.2 v1:172.21.15.104:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f55cc001670 con 0x7f55d80a9930 2024-03-29T11:20:11.527 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.548+0000 7f55d5ffb640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:20:11.528 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.549+0000 7f55dd3c4640 1 -- 172.21.15.104:0/3479485160 shutdown_connections 2024-03-29T11:20:11.528 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.549+0000 7f55dd3c4640 1 -- 172.21.15.104:0/3479485160 >> 172.21.15.104:0/3479485160 conn(0x7f55d80b35e0 msgr2=0x7f55d80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:11.528 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.549+0000 7f55dd3c4640 1 -- 172.21.15.104:0/3479485160 shutdown_connections 2024-03-29T11:20:11.528 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.549+0000 7f55dd3c4640 1 -- 172.21.15.104:0/3479485160 wait complete. 2024-03-29T11:20:11.528 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.550+0000 7f55dd3c4640 10 monclient: build_initial_monmap 2024-03-29T11:20:11.528 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.550+0000 7f55dd3c4640 10 monclient: monmap: 2024-03-29T11:20:11.529 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:20:11.529 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:11.529 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:20:11.529 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:20:11.529 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:11.529 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:20:11.529 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:20:11.529 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:20:11.529 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:20:11.529 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:20:11.529 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.550+0000 7f55dd3c4640 1 librados: starting msgr at 2024-03-29T11:20:11.529 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.550+0000 7f55dd3c4640 1 librados: starting objecter 2024-03-29T11:20:11.529 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.550+0000 7f55dd3c4640 1 Processor -- start 2024-03-29T11:20:11.530 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.551+0000 7f55dd3c4640 1 -- start start 2024-03-29T11:20:11.530 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.551+0000 7f55dd3c4640 1 librados: setting wanted keys 2024-03-29T11:20:11.530 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.551+0000 7f55dd3c4640 1 librados: calling monclient init 2024-03-29T11:20:11.530 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.551+0000 7f55dd3c4640 10 monclient: init 2024-03-29T11:20:11.530 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.551+0000 7f55dd3c4640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:11.530 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.551+0000 7f55dd3c4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:11.530 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.551+0000 7f55dd3c4640 10 monclient(hunting): picked mon.noname-a con 0x7f55d80a9930 addr v1:172.21.15.104:6789/0 2024-03-29T11:20:11.530 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.551+0000 7f55dd3c4640 10 monclient(hunting): picked mon.noname-b con 0x7f55d8149dc0 addr v1:172.21.15.119:6789/0 2024-03-29T11:20:11.530 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.551+0000 7f55dd3c4640 10 monclient(hunting): picked mon.noname-c con 0x7f55d814ff40 addr v1:172.21.15.104:6790/0 2024-03-29T11:20:11.530 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.551+0000 7f55dd3c4640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55d813dbe0 con 0x7f55d80a9930 2024-03-29T11:20:11.530 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.551+0000 7f55dd3c4640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55d806e9f0 con 0x7f55d8149dc0 2024-03-29T11:20:11.530 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.551+0000 7f55dd3c4640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55d81387b0 con 0x7f55d814ff40 2024-03-29T11:20:11.530 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.551+0000 7f55dd3c4640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:11.531 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.551+0000 7f55dd3c4640 10 monclient(hunting): authenticate will time out at 992.802876s 2024-03-29T11:20:11.538 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.560+0000 7f55d77fe640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f55d814ff40 0x7f55d82000a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:56893/0 (socket says 172.21.15.104:56893) 2024-03-29T11:20:11.538 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.560+0000 7f55d77fe640 1 -- 172.21.15.104:0/799326608 learned_addr learned my addr 172.21.15.104:0/799326608 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:20:11.538 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.560+0000 7f55bf7fe640 10 client.?.objecter ms_handle_connect 0x7f55d80a9930 2024-03-29T11:20:11.538 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.560+0000 7f55bf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:11.538 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.560+0000 7f55bf7fe640 1 -- 172.21.15.104:0/799326608 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4190519565 0 0) 0x7f55d813dbe0 con 0x7f55d80a9930 2024-03-29T11:20:11.538 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.560+0000 7f55bf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:11.538 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.560+0000 7f55bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:11.538 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.560+0000 7f55bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:11.538 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.560+0000 7f55bf7fe640 10 monclient(hunting): my global_id is 4263 2024-03-29T11:20:11.539 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.560+0000 7f55bf7fe640 1 -- 172.21.15.104:0/799326608 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55c4004eb0 con 0x7f55d80a9930 2024-03-29T11:20:11.539 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.560+0000 7f55bf7fe640 1 -- 172.21.15.104:0/799326608 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1882828686 0 0) 0x7f55c4004eb0 con 0x7f55d80a9930 2024-03-29T11:20:11.539 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.561+0000 7f55bf7fe640 1 -- 172.21.15.104:0/799326608 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f55d813dbe0 con 0x7f55d80a9930 2024-03-29T11:20:11.539 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.561+0000 7f55bf7fe640 1 -- 172.21.15.104:0/799326608 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f55cc001d80 con 0x7f55d80a9930 2024-03-29T11:20:11.539 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.561+0000 7f55bf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:11.539 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.561+0000 7f55bf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:11.539 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.561+0000 7f55bf7fe640 10 monclient(hunting): dump: 2024-03-29T11:20:11.539 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:20:11.539 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:20:11.539 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:11.539 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:11.539 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:11.540 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:20:11.540 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:20:11.540 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:20:11.540 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:20:11.540 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:20:11.540 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.561+0000 7f55bf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:11.540 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.561+0000 7f55bf7fe640 10 client.?.objecter ms_handle_connect 0x7f55d814ff40 2024-03-29T11:20:11.540 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.561+0000 7f55bf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:11.540 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.561+0000 7f55bf7fe640 1 -- 172.21.15.104:0/799326608 <== mon.0 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1979956996 0 0) 0x7f55d813dbe0 con 0x7f55d80a9930 2024-03-29T11:20:11.540 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.561+0000 7f55bf7fe640 1 -- 172.21.15.104:0/799326608 >> v1:172.21.15.104:6790/0 conn(0x7f55d814ff40 legacy=0x7f55d82000a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:11.540 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.561+0000 7f55bf7fe640 1 -- 172.21.15.104:0/799326608 >> v1:172.21.15.119:6789/0 conn(0x7f55d8149dc0 legacy=0x7f55d813f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:11.540 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.561+0000 7f55bf7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:20:11.542 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.561+0000 7f55bf7fe640 1 monclient: found mon.a 2024-03-29T11:20:11.543 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.561+0000 7f55bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:11.543 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.561+0000 7f55bf7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:20:11.543 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.561+0000 7f55bf7fe640 1 -- 172.21.15.104:0/799326608 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55d8138940 con 0x7f55d80a9930 2024-03-29T11:20:11.543 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.561+0000 7f55bf7fe640 10 monclient: _finish_auth 0 2024-03-29T11:20:11.543 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.561+0000 7f55bf7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:20:11.543 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.561+0000 7f55bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:11.543 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.561+0000 7f55bf7fe640 10 client.?.objecter ms_handle_connect 0x7f55d8149dc0 2024-03-29T11:20:11.543 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.561+0000 7f55bf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:11.543 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.561+0000 7f55dd3c4640 5 monclient: authenticate success, global_id 4263 2024-03-29T11:20:11.543 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.561+0000 7f55dd3c4640 10 monclient: _renew_subs 2024-03-29T11:20:11.545 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.561+0000 7f55dd3c4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:20:11.545 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.561+0000 7f55dd3c4640 1 -- 172.21.15.104:0/799326608 --> v1:172.21.15.104:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55d813c650 con 0x7f55d80a9930 2024-03-29T11:20:11.545 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.561+0000 7f55dd3c4640 10 client.4263.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:11.545 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.561+0000 7f55dd3c4640 10 monclient: _renew_subs 2024-03-29T11:20:11.545 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.561+0000 7f55dd3c4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:20:11.548 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.561+0000 7f55dd3c4640 1 -- 172.21.15.104:0/799326608 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f55d806ee80 con 0x7f55d80a9930 2024-03-29T11:20:11.548 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.561+0000 7f55dd3c4640 1 librados: init done 2024-03-29T11:20:11.548 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.562+0000 7f55dd3c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:11.548 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.562+0000 7f55dd3c4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:11.548 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.562+0000 7f55dd3c4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:20:11.548 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.562+0000 7f55dd3c4640 1 -- 172.21.15.104:0/799326608 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55d8075f10 con 0x7f55d80a9930 2024-03-29T11:20:11.548 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.567+0000 7f55bf7fe640 1 -- 172.21.15.104:0/799326608 <== mon.0 v1:172.21.15.104:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f55cc0011f0 con 0x7f55d80a9930 2024-03-29T11:20:11.549 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.567+0000 7f55bf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:11.549 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.567+0000 7f55bf7fe640 1 -- 172.21.15.104:0/799326608 <== mon.0 v1:172.21.15.104:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f55cc003b40 con 0x7f55d80a9930 2024-03-29T11:20:11.549 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.567+0000 7f55bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:11.549 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.567+0000 7f55bf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:11.549 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.567+0000 7f55bf7fe640 10 monclient: dump: 2024-03-29T11:20:11.549 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:20:11.549 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:20:11.549 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:11.549 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:11.549 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:11.549 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:20:11.549 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:20:11.549 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:20:11.550 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:20:11.550 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:20:11.550 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.567+0000 7f55bf7fe640 1 -- 172.21.15.104:0/799326608 <== mon.0 v1:172.21.15.104:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f55cc003d60 con 0x7f55d80a9930 2024-03-29T11:20:11.550 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.567+0000 7f55bf7fe640 1 -- 172.21.15.104:0/799326608 <== mon.0 v1:172.21.15.104:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f55cc083b20 con 0x7f55d80a9930 2024-03-29T11:20:11.550 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.567+0000 7f55bf7fe640 10 client.4263.objecter ms_dispatch 0x7f55d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:11.550 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.567+0000 7f55bf7fe640 3 client.4263.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:11.550 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.567+0000 7f55bf7fe640 3 client.4263.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:11.550 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.568+0000 7f55bf7fe640 20 client.4263.objecter dump_active .. 0 homeless 2024-03-29T11:20:11.550 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.568+0000 7f55d4ff9640 4 set_mon_vals no callback set 2024-03-29T11:20:11.550 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.568+0000 7f55d67fc640 10 monclient: get_auth_request con 0x7f55c406dc70 auth_method 0 2024-03-29T11:20:11.550 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.569+0000 7f55bf7fe640 10 client.4263.objecter ms_handle_connect 0x7f55c406dc70 2024-03-29T11:20:11.551 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.574+0000 7f55bf7fe640 1 -- 172.21.15.104:0/799326608 <== mon.0 v1:172.21.15.104:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f55cc04f480 con 0x7f55d80a9930 2024-03-29T11:20:11.551 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.574+0000 7f55bf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:11.551 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.574+0000 7f55bf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:11.872 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.895+0000 7f55dd3c4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:11.872 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.895+0000 7f55dd3c4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:11.872 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.895+0000 7f55dd3c4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:20:11.872 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.895+0000 7f55dd3c4640 1 -- 172.21.15.104:0/799326608 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f55d813d100 con 0x7f55d80a9930 2024-03-29T11:20:11.873 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.896+0000 7f55bf7fe640 1 -- 172.21.15.104:0/799326608 <== mon.0 v1:172.21.15.104:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f55cc0513b0 con 0x7f55d80a9930 2024-03-29T11:20:11.873 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.896+0000 7f55bf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:11.873 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.896+0000 7f55bf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:11.873 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:20:11.873 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:11.875 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.899+0000 7f55dd3c4640 10 librados: watch_flush enter 2024-03-29T11:20:11.876 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.899+0000 7f55dd3c4640 10 librados: watch_flush exit 2024-03-29T11:20:11.876 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.899+0000 7f55dd3c4640 20 client.4263.objecter shutdown clearing up homeless session... 2024-03-29T11:20:11.876 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.899+0000 7f55dd3c4640 10 client.4263.objecter successfully canceled tick 2024-03-29T11:20:11.878 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.902+0000 7f55dd3c4640 1 -- 172.21.15.104:0/799326608 >> v1:172.21.15.104:6816/3341126660 conn(0x7f55c406dc70 legacy=0x7f55c4070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:11.879 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.902+0000 7f55dd3c4640 10 monclient: shutdown 2024-03-29T11:20:11.879 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.902+0000 7f55dd3c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:11.879 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.902+0000 7f55dd3c4640 1 -- 172.21.15.104:0/799326608 >> v1:172.21.15.104:6789/0 conn(0x7f55d80a9930 legacy=0x7f55d8137650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:11.879 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.902+0000 7f55dd3c4640 1 -- 172.21.15.104:0/799326608 shutdown_connections 2024-03-29T11:20:11.879 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.902+0000 7f55dd3c4640 1 -- 172.21.15.104:0/799326608 >> 172.21.15.104:0/799326608 conn(0x7f55d80b35e0 msgr2=0x7f55d80b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:11.879 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.902+0000 7f55dd3c4640 1 -- 172.21.15.104:0/799326608 shutdown_connections 2024-03-29T11:20:11.879 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.903+0000 7f55dd3c4640 1 -- 172.21.15.104:0/799326608 wait complete. 2024-03-29T11:20:11.880 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:11.903+0000 7f55dd3c4640 1 librados: shutdown 2024-03-29T11:20:11.898 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:18.705 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:18.854 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.876+0000 7fa074b4b640 10 monclient: get_monmap_and_config 2024-03-29T11:20:18.855 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.876+0000 7fa074b4b640 10 monclient: build_initial_monmap 2024-03-29T11:20:18.855 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.876+0000 7fa074b4b640 10 monclient: monmap: 2024-03-29T11:20:18.855 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:20:18.855 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:20:18.855 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:20:18.877679+0000 2024-03-29T11:20:18.855 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:20:18.877679+0000 2024-03-29T11:20:18.855 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:18.855 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:20:18.855 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:20:18.855 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:20:18.855 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:20:18.855 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:20:18.856 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.878+0000 7fa074b4b640 1 Processor -- start 2024-03-29T11:20:18.856 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.878+0000 7fa074b4b640 1 -- start start 2024-03-29T11:20:18.856 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.878+0000 7fa074b4b640 10 monclient: init 2024-03-29T11:20:18.857 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.879+0000 7fa074b4b640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:18.857 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.879+0000 7fa074b4b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:18.857 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.879+0000 7fa074b4b640 10 monclient(hunting): picked mon.noname-a con 0x7fa0700ab360 addr v1:172.21.15.104:6789/0 2024-03-29T11:20:18.857 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.879+0000 7fa074b4b640 10 monclient(hunting): picked mon.noname-c con 0x7fa07014fb00 addr v1:172.21.15.119:6789/0 2024-03-29T11:20:18.858 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.879+0000 7fa074b4b640 10 monclient(hunting): picked mon.noname-b con 0x7fa070153090 addr v1:172.21.15.104:6790/0 2024-03-29T11:20:18.858 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.879+0000 7fa074b4b640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0700a94e0 con 0x7fa0700ab360 2024-03-29T11:20:18.858 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.879+0000 7fa074b4b640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0700afb60 con 0x7fa07014fb00 2024-03-29T11:20:18.858 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.879+0000 7fa074b4b640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0700aab80 con 0x7fa070153090 2024-03-29T11:20:18.858 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.879+0000 7fa074b4b640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:18.858 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.879+0000 7fa074b4b640 10 monclient(hunting): authenticate will time out at 1000.130324s 2024-03-29T11:20:18.859 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.883+0000 7fa06e59c640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fa0700ab360 0x7fa0700ad450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:36277/0 (socket says 172.21.15.104:36277) 2024-03-29T11:20:18.859 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.883+0000 7fa06e59c640 1 -- 172.21.15.104:0/4272075696 learned_addr learned my addr 172.21.15.104:0/4272075696 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:20:18.863 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.887+0000 7fa06d59a640 1 -- 172.21.15.104:0/4272075696 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3819024909 0 0) 0x7fa0700a94e0 con 0x7fa0700ab360 2024-03-29T11:20:18.863 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.887+0000 7fa06d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:20:18.863 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.887+0000 7fa06d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:18.863 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.887+0000 7fa06d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:18.863 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.887+0000 7fa06d59a640 10 monclient(hunting): my global_id is 4266 2024-03-29T11:20:18.863 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.887+0000 7fa06d59a640 1 -- 172.21.15.104:0/4272075696 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa058003520 con 0x7fa0700ab360 2024-03-29T11:20:18.863 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.887+0000 7fa06d59a640 1 -- 172.21.15.104:0/4272075696 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 876401368 0 0) 0x7fa0700aab80 con 0x7fa070153090 2024-03-29T11:20:18.864 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.887+0000 7fa06d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:20:18.864 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.887+0000 7fa06d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:18.864 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.887+0000 7fa06d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:18.864 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.887+0000 7fa06d59a640 10 monclient(hunting): my global_id is 4268 2024-03-29T11:20:18.864 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.887+0000 7fa06d59a640 1 -- 172.21.15.104:0/4272075696 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0700a94e0 con 0x7fa070153090 2024-03-29T11:20:18.864 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.887+0000 7fa06d59a640 1 -- 172.21.15.104:0/4272075696 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 424793449 0 0) 0x7fa058003520 con 0x7fa0700ab360 2024-03-29T11:20:18.864 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.887+0000 7fa06d59a640 1 -- 172.21.15.104:0/4272075696 >> v1:172.21.15.104:6790/0 conn(0x7fa070153090 legacy=0x7fa07015cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:18.864 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.887+0000 7fa06d59a640 1 -- 172.21.15.104:0/4272075696 >> v1:172.21.15.119:6789/0 conn(0x7fa07014fb00 legacy=0x7fa070152980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:18.864 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.887+0000 7fa06d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:20:18.864 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.887+0000 7fa06d59a640 1 monclient: found mon.noname-a 2024-03-29T11:20:18.865 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.887+0000 7fa06d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:18.865 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.887+0000 7fa06d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.104:6789/0 2024-03-29T11:20:18.865 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.887+0000 7fa06d59a640 1 -- 172.21.15.104:0/4272075696 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0701568d0 con 0x7fa0700ab360 2024-03-29T11:20:18.865 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.887+0000 7fa06d59a640 10 monclient: _finish_auth 0 2024-03-29T11:20:18.865 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.887+0000 7fa06d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:20:18.865 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.887+0000 7fa06d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:18.865 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.887+0000 7fa06d59a640 1 -- 172.21.15.104:0/4272075696 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fa064001690 con 0x7fa0700ab360 2024-03-29T11:20:18.865 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.887+0000 7fa06d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:18.865 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.887+0000 7fa06d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:18.865 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.887+0000 7fa06d59a640 10 monclient: dump: 2024-03-29T11:20:18.865 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:20:18.865 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:20:18.865 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:18.865 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:18.866 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:18.866 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:20:18.866 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:20:18.866 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:20:18.866 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:20:18.866 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:20:18.866 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.887+0000 7fa074b4b640 5 monclient: authenticate success, global_id 4266 2024-03-29T11:20:18.866 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.887+0000 7fa074b4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:18.866 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.887+0000 7fa06d59a640 1 -- 172.21.15.104:0/4272075696 <== mon.0 v1:172.21.15.104:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0640017f0 con 0x7fa0700ab360 2024-03-29T11:20:18.866 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.887+0000 7fa06d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:18.866 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.887+0000 7fa074b4b640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:18.866 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.887+0000 7fa06d59a640 1 -- 172.21.15.104:0/4272075696 <== mon.0 v1:172.21.15.104:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fa064001c00 con 0x7fa0700ab360 2024-03-29T11:20:18.866 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.887+0000 7fa06d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:18.866 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.887+0000 7fa074b4b640 4 set_mon_vals no callback set 2024-03-29T11:20:18.867 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.887+0000 7fa06d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:18.867 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.887+0000 7fa06d59a640 10 monclient: dump: 2024-03-29T11:20:18.867 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:20:18.867 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:20:18.867 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:18.867 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:18.867 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:18.867 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:20:18.867 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:20:18.867 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:20:18.867 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:20:18.867 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:20:18.868 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.888+0000 7fa074b4b640 10 monclient: shutdown 2024-03-29T11:20:18.869 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.888+0000 7fa074b4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:18.869 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.888+0000 7fa074b4b640 1 -- 172.21.15.104:0/4272075696 >> v1:172.21.15.104:6789/0 conn(0x7fa0700ab360 legacy=0x7fa0700ad450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:18.869 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.889+0000 7fa074b4b640 1 -- 172.21.15.104:0/4272075696 shutdown_connections 2024-03-29T11:20:18.869 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.889+0000 7fa074b4b640 1 -- 172.21.15.104:0/4272075696 >> 172.21.15.104:0/4272075696 conn(0x7fa0700b35e0 msgr2=0x7fa0700b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:18.869 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.891+0000 7fa074b4b640 1 -- 172.21.15.104:0/4272075696 shutdown_connections 2024-03-29T11:20:18.869 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.891+0000 7fa074b4b640 1 -- 172.21.15.104:0/4272075696 wait complete. 2024-03-29T11:20:18.869 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.892+0000 7fa074b4b640 10 monclient: build_initial_monmap 2024-03-29T11:20:18.869 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.892+0000 7fa074b4b640 10 monclient: monmap: 2024-03-29T11:20:18.869 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:20:18.869 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:18.869 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:20:18.869 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:20:18.869 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:18.870 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:20:18.870 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:20:18.870 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:20:18.870 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:20:18.870 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:20:18.874 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.892+0000 7fa074b4b640 1 librados: starting msgr at 2024-03-29T11:20:18.874 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.892+0000 7fa074b4b640 1 librados: starting objecter 2024-03-29T11:20:18.874 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.892+0000 7fa074b4b640 1 Processor -- start 2024-03-29T11:20:18.874 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.892+0000 7fa074b4b640 1 -- start start 2024-03-29T11:20:18.874 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.892+0000 7fa074b4b640 1 librados: setting wanted keys 2024-03-29T11:20:18.874 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.892+0000 7fa074b4b640 1 librados: calling monclient init 2024-03-29T11:20:18.874 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.892+0000 7fa074b4b640 10 monclient: init 2024-03-29T11:20:18.875 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.893+0000 7fa074b4b640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:18.875 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.893+0000 7fa074b4b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:20:18.875 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.893+0000 7fa074b4b640 10 monclient(hunting): picked mon.noname-b con 0x7fa0700ab360 addr v1:172.21.15.119:6789/0 2024-03-29T11:20:18.875 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.893+0000 7fa074b4b640 10 monclient(hunting): picked mon.noname-a con 0x7fa07014fb00 addr v1:172.21.15.104:6789/0 2024-03-29T11:20:18.875 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.893+0000 7fa074b4b640 10 monclient(hunting): picked mon.noname-c con 0x7fa070153090 addr v1:172.21.15.104:6790/0 2024-03-29T11:20:18.875 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.893+0000 7fa074b4b640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0701564a0 con 0x7fa07014fb00 2024-03-29T11:20:18.875 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.893+0000 7fa074b4b640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0700881e0 con 0x7fa0700ab360 2024-03-29T11:20:18.875 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.893+0000 7fa074b4b640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0701506d0 con 0x7fa070153090 2024-03-29T11:20:18.875 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.893+0000 7fa074b4b640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:18.875 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.893+0000 7fa074b4b640 10 monclient(hunting): authenticate will time out at 1000.144726s 2024-03-29T11:20:18.875 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.896+0000 7fa06ed9d640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7fa070153090 0x7fa0701f7c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:51925/0 (socket says 172.21.15.104:51925) 2024-03-29T11:20:18.875 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.896+0000 7fa06ed9d640 1 -- 172.21.15.104:0/3019345440 learned_addr learned my addr 172.21.15.104:0/3019345440 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:20:18.879 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.900+0000 7fa05effd640 10 client.?.objecter ms_handle_connect 0x7fa070153090 2024-03-29T11:20:18.879 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.900+0000 7fa05effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:18.879 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.901+0000 7fa05effd640 1 -- 172.21.15.104:0/3019345440 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 8504806 0 0) 0x7fa0701506d0 con 0x7fa070153090 2024-03-29T11:20:18.879 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.901+0000 7fa05effd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:18.879 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.901+0000 7fa05effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:18.879 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.901+0000 7fa05effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:18.879 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.901+0000 7fa05effd640 10 monclient(hunting): my global_id is 4271 2024-03-29T11:20:18.879 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.901+0000 7fa05effd640 1 -- 172.21.15.104:0/3019345440 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa050004eb0 con 0x7fa070153090 2024-03-29T11:20:18.879 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.901+0000 7fa05effd640 10 client.?.objecter ms_handle_connect 0x7fa07014fb00 2024-03-29T11:20:18.879 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.901+0000 7fa05effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:18.879 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.901+0000 7fa05effd640 1 -- 172.21.15.104:0/3019345440 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1912401104 0 0) 0x7fa050004eb0 con 0x7fa070153090 2024-03-29T11:20:18.879 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.901+0000 7fa05effd640 1 -- 172.21.15.104:0/3019345440 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0701506d0 con 0x7fa070153090 2024-03-29T11:20:18.879 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.901+0000 7fa05effd640 1 -- 172.21.15.104:0/3019345440 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fa0600018f0 con 0x7fa070153090 2024-03-29T11:20:18.879 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.902+0000 7fa05effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:18.880 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.902+0000 7fa05effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:18.880 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.902+0000 7fa05effd640 10 monclient(hunting): dump: 2024-03-29T11:20:18.880 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:20:18.880 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:20:18.880 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:18.880 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:18.880 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:18.880 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:20:18.880 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:20:18.880 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:20:18.880 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:20:18.880 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:20:18.880 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.902+0000 7fa05effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:18.880 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.902+0000 7fa05effd640 1 -- 172.21.15.104:0/3019345440 <== mon.2 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 815247157 0 0) 0x7fa0701506d0 con 0x7fa070153090 2024-03-29T11:20:18.880 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.902+0000 7fa05effd640 1 -- 172.21.15.104:0/3019345440 >> v1:172.21.15.119:6789/0 conn(0x7fa0700ab360 legacy=0x7fa0700873c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:20:18.883 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.903+0000 7fa05effd640 1 -- 172.21.15.104:0/3019345440 >> v1:172.21.15.104:6789/0 conn(0x7fa07014fb00 legacy=0x7fa070087ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:18.883 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.903+0000 7fa05effd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:18.883 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.903+0000 7fa05effd640 1 monclient: found mon.c 2024-03-29T11:20:18.883 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.903+0000 7fa05effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:18.883 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.903+0000 7fa05effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:20:18.883 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.903+0000 7fa05effd640 1 -- 172.21.15.104:0/3019345440 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0701554a0 con 0x7fa070153090 2024-03-29T11:20:18.883 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.903+0000 7fa05effd640 10 monclient: _finish_auth 0 2024-03-29T11:20:18.883 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.903+0000 7fa05effd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:18.883 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.903+0000 7fa05effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:18.883 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.903+0000 7fa074b4b640 5 monclient: authenticate success, global_id 4271 2024-03-29T11:20:18.883 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.903+0000 7fa074b4b640 10 monclient: _renew_subs 2024-03-29T11:20:18.883 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.903+0000 7fa074b4b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:20:18.883 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.903+0000 7fa074b4b640 1 -- 172.21.15.104:0/3019345440 --> v1:172.21.15.104:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa070071840 con 0x7fa070153090 2024-03-29T11:20:18.883 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.904+0000 7fa074b4b640 10 client.4271.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:18.884 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.904+0000 7fa074b4b640 10 monclient: _renew_subs 2024-03-29T11:20:18.884 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.904+0000 7fa074b4b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:20:18.884 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.904+0000 7fa074b4b640 1 -- 172.21.15.104:0/3019345440 --> v1:172.21.15.104:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa07008ae50 con 0x7fa070153090 2024-03-29T11:20:18.884 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.904+0000 7fa074b4b640 1 librados: init done 2024-03-29T11:20:18.884 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.905+0000 7fa074b4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:18.884 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.905+0000 7fa074b4b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:18.884 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.906+0000 7fa074b4b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:20:18.884 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.906+0000 7fa074b4b640 1 -- 172.21.15.104:0/3019345440 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa07008bc00 con 0x7fa070153090 2024-03-29T11:20:18.885 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.907+0000 7fa05effd640 1 -- 172.21.15.104:0/3019345440 <== mon.2 v1:172.21.15.104:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa060001ac0 con 0x7fa070153090 2024-03-29T11:20:18.885 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.907+0000 7fa05effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:18.885 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.907+0000 7fa05effd640 1 -- 172.21.15.104:0/3019345440 <== mon.2 v1:172.21.15.104:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fa0600038c0 con 0x7fa070153090 2024-03-29T11:20:18.885 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.907+0000 7fa05effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:18.885 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.907+0000 7fa05effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:18.885 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.907+0000 7fa05effd640 10 monclient: dump: 2024-03-29T11:20:18.885 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:20:18.885 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:20:18.885 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:18.885 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:18.885 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:18.885 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:20:18.886 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:20:18.886 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:20:18.886 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:20:18.886 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:20:18.886 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.907+0000 7fa06cd99640 4 set_mon_vals no callback set 2024-03-29T11:20:18.886 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.909+0000 7fa05effd640 1 -- 172.21.15.104:0/3019345440 <== mon.2 v1:172.21.15.104:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7fa060003a20 con 0x7fa070153090 2024-03-29T11:20:18.887 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.910+0000 7fa05effd640 1 -- 172.21.15.104:0/3019345440 <== mon.2 v1:172.21.15.104:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7fa060084a00 con 0x7fa070153090 2024-03-29T11:20:18.887 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.910+0000 7fa05effd640 10 client.4271.objecter ms_dispatch 0x7fa070000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:18.887 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.910+0000 7fa05effd640 3 client.4271.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:18.887 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.910+0000 7fa05effd640 3 client.4271.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:18.887 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.910+0000 7fa05effd640 20 client.4271.objecter dump_active .. 0 homeless 2024-03-29T11:20:18.887 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.910+0000 7fa06e59c640 10 monclient: get_auth_request con 0x7fa05006dc80 auth_method 0 2024-03-29T11:20:18.887 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.910+0000 7fa05effd640 10 client.4271.objecter ms_handle_connect 0x7fa05006dc80 2024-03-29T11:20:18.895 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.919+0000 7fa05effd640 1 -- 172.21.15.104:0/3019345440 <== mon.2 v1:172.21.15.104:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa060050110 con 0x7fa070153090 2024-03-29T11:20:18.895 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.919+0000 7fa05effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:18.895 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:18.919+0000 7fa05effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:19.223 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:19.246+0000 7fa074b4b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:19.223 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:19.246+0000 7fa074b4b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:19.223 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:19.246+0000 7fa074b4b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:20:19.223 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:19.246+0000 7fa074b4b640 1 -- 172.21.15.104:0/3019345440 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0700b39c0 con 0x7fa070153090 2024-03-29T11:20:19.223 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:19.247+0000 7fa05effd640 1 -- 172.21.15.104:0/3019345440 <== mon.2 v1:172.21.15.104:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7fa060052210 con 0x7fa070153090 2024-03-29T11:20:19.223 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:19.247+0000 7fa05effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:19.223 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:19.247+0000 7fa05effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:19.227 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:20:19.227 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:19.230 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:19.253+0000 7fa074b4b640 10 librados: watch_flush enter 2024-03-29T11:20:19.230 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:19.253+0000 7fa074b4b640 10 librados: watch_flush exit 2024-03-29T11:20:19.231 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:19.253+0000 7fa074b4b640 20 client.4271.objecter shutdown clearing up homeless session... 2024-03-29T11:20:19.231 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:19.253+0000 7fa074b4b640 10 client.4271.objecter successfully canceled tick 2024-03-29T11:20:19.231 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:19.254+0000 7fa074b4b640 1 -- 172.21.15.104:0/3019345440 >> v1:172.21.15.104:6816/3341126660 conn(0x7fa05006dc80 legacy=0x7fa050070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:19.231 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:19.254+0000 7fa074b4b640 10 monclient: shutdown 2024-03-29T11:20:19.231 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:19.254+0000 7fa074b4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:19.231 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:19.254+0000 7fa074b4b640 1 -- 172.21.15.104:0/3019345440 >> v1:172.21.15.104:6790/0 conn(0x7fa070153090 legacy=0x7fa0701f7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:19.233 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:19.256+0000 7fa074b4b640 1 -- 172.21.15.104:0/3019345440 shutdown_connections 2024-03-29T11:20:19.233 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:19.257+0000 7fa074b4b640 1 -- 172.21.15.104:0/3019345440 >> 172.21.15.104:0/3019345440 conn(0x7fa0700b35e0 msgr2=0x7fa070162d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:19.233 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:19.257+0000 7fa074b4b640 1 -- 172.21.15.104:0/3019345440 shutdown_connections 2024-03-29T11:20:19.234 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:19.257+0000 7fa074b4b640 1 -- 172.21.15.104:0/3019345440 wait complete. 2024-03-29T11:20:19.234 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:19.257+0000 7fa074b4b640 1 librados: shutdown 2024-03-29T11:20:19.253 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:26.060 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:26.208 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.231+0000 7fba5b243640 10 monclient: get_monmap_and_config 2024-03-29T11:20:26.208 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.231+0000 7fba5b243640 10 monclient: build_initial_monmap 2024-03-29T11:20:26.209 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.231+0000 7fba5b243640 10 monclient: monmap: 2024-03-29T11:20:26.209 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:20:26.209 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:20:26.209 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:20:26.232621+0000 2024-03-29T11:20:26.209 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:20:26.232621+0000 2024-03-29T11:20:26.209 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:26.209 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:20:26.209 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:20:26.209 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:20:26.209 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:20:26.209 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:20:26.210 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.232+0000 7fba5b243640 1 Processor -- start 2024-03-29T11:20:26.210 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.232+0000 7fba5b243640 1 -- start start 2024-03-29T11:20:26.210 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.232+0000 7fba5b243640 10 monclient: init 2024-03-29T11:20:26.210 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.233+0000 7fba5b243640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:26.210 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.233+0000 7fba5b243640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:20:26.211 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.233+0000 7fba5b243640 10 monclient(hunting): picked mon.noname-b con 0x7fba54071de0 addr v1:172.21.15.104:6790/0 2024-03-29T11:20:26.216 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.237+0000 7fba5b243640 10 monclient(hunting): picked mon.noname-a con 0x7fba540721c0 addr v1:172.21.15.104:6789/0 2024-03-29T11:20:26.216 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.238+0000 7fba53fff640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fba540721c0 0x7fba54145670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59023/0 (socket says 172.21.15.104:59023) 2024-03-29T11:20:26.216 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.238+0000 7fba53fff640 1 -- 172.21.15.104:0/3967328602 learned_addr learned my addr 172.21.15.104:0/3967328602 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:20:26.216 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.238+0000 7fba5b243640 10 monclient(hunting): picked mon.noname-c con 0x7fba541425d0 addr v1:172.21.15.119:6789/0 2024-03-29T11:20:26.216 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.238+0000 7fba5b243640 1 -- 172.21.15.104:0/3967328602 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba5413fb10 con 0x7fba540721c0 2024-03-29T11:20:26.216 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.238+0000 7fba5b243640 1 -- 172.21.15.104:0/3967328602 --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba54142c20 con 0x7fba541425d0 2024-03-29T11:20:26.216 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.238+0000 7fba5b243640 1 -- 172.21.15.104:0/3967328602 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba540ada50 con 0x7fba54071de0 2024-03-29T11:20:26.216 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.238+0000 7fba5b243640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:26.216 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.238+0000 7fba5b243640 10 monclient(hunting): authenticate will time out at 1007.489458s 2024-03-29T11:20:26.216 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.239+0000 7fba537fe640 1 -- 172.21.15.104:0/3967328602 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1410506030 0 0) 0x7fba5413fb10 con 0x7fba540721c0 2024-03-29T11:20:26.217 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.239+0000 7fba537fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:26.217 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.239+0000 7fba537fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:26.217 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.239+0000 7fba537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:26.217 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.239+0000 7fba537fe640 10 monclient(hunting): my global_id is 4272 2024-03-29T11:20:26.217 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.239+0000 7fba537fe640 1 -- 172.21.15.104:0/3967328602 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba3c003520 con 0x7fba540721c0 2024-03-29T11:20:26.217 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.239+0000 7fba537fe640 1 -- 172.21.15.104:0/3967328602 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3619417975 0 0) 0x7fba3c003520 con 0x7fba540721c0 2024-03-29T11:20:26.217 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.239+0000 7fba537fe640 1 -- 172.21.15.104:0/3967328602 >> v1:172.21.15.104:6790/0 conn(0x7fba54071de0 legacy=0x7fba5413f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:26.217 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.239+0000 7fba537fe640 1 -- 172.21.15.104:0/3967328602 >> v1:172.21.15.119:6789/0 conn(0x7fba541425d0 legacy=0x7fba5414da10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:20:26.221 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.240+0000 7fba537fe640 10 monclient: _finish_hunting 0 2024-03-29T11:20:26.221 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.240+0000 7fba537fe640 1 monclient: found mon.noname-a 2024-03-29T11:20:26.221 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.240+0000 7fba537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:26.221 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.240+0000 7fba537fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.104:6789/0 2024-03-29T11:20:26.221 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.240+0000 7fba537fe640 1 -- 172.21.15.104:0/3967328602 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba540b0500 con 0x7fba540721c0 2024-03-29T11:20:26.221 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.240+0000 7fba537fe640 10 monclient: _finish_auth 0 2024-03-29T11:20:26.221 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.240+0000 7fba537fe640 10 monclient: _check_auth_tickets 2024-03-29T11:20:26.221 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.240+0000 7fba537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:26.221 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.240+0000 7fba537fe640 1 -- 172.21.15.104:0/3967328602 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fba48001670 con 0x7fba540721c0 2024-03-29T11:20:26.221 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.240+0000 7fba537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:26.221 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.240+0000 7fba537fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:26.221 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.240+0000 7fba537fe640 10 monclient: dump: 2024-03-29T11:20:26.222 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:20:26.222 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:20:26.222 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:26.222 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:26.222 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:26.222 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:20:26.222 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:20:26.222 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:20:26.222 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:20:26.222 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:20:26.222 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.240+0000 7fba5b243640 5 monclient: authenticate success, global_id 4272 2024-03-29T11:20:26.222 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.240+0000 7fba5b243640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:26.222 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.240+0000 7fba537fe640 1 -- 172.21.15.104:0/3967328602 <== mon.0 v1:172.21.15.104:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba48001900 con 0x7fba540721c0 2024-03-29T11:20:26.222 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.240+0000 7fba537fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:26.222 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.240+0000 7fba537fe640 1 -- 172.21.15.104:0/3967328602 <== mon.0 v1:172.21.15.104:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fba48001cd0 con 0x7fba540721c0 2024-03-29T11:20:26.223 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.240+0000 7fba537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:26.223 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.240+0000 7fba537fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:26.223 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.240+0000 7fba537fe640 10 monclient: dump: 2024-03-29T11:20:26.223 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:20:26.223 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:20:26.223 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:26.223 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:26.223 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:26.223 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:20:26.223 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:20:26.223 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:20:26.223 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:20:26.223 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:20:26.223 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.240+0000 7fba5b243640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:26.223 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.240+0000 7fba5b243640 4 set_mon_vals no callback set 2024-03-29T11:20:26.224 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.241+0000 7fba5b243640 10 monclient: shutdown 2024-03-29T11:20:26.224 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.241+0000 7fba5b243640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:26.224 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.241+0000 7fba5b243640 1 -- 172.21.15.104:0/3967328602 >> v1:172.21.15.104:6789/0 conn(0x7fba540721c0 legacy=0x7fba54145670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:26.224 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.242+0000 7fba5b243640 1 -- 172.21.15.104:0/3967328602 shutdown_connections 2024-03-29T11:20:26.224 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.242+0000 7fba5b243640 1 -- 172.21.15.104:0/3967328602 >> 172.21.15.104:0/3967328602 conn(0x7fba5406b660 msgr2=0x7fba5406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:26.224 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.242+0000 7fba5b243640 1 -- 172.21.15.104:0/3967328602 shutdown_connections 2024-03-29T11:20:26.224 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.242+0000 7fba5b243640 1 -- 172.21.15.104:0/3967328602 wait complete. 2024-03-29T11:20:26.224 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.243+0000 7fba5b243640 10 monclient: build_initial_monmap 2024-03-29T11:20:26.224 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.243+0000 7fba5b243640 10 monclient: monmap: 2024-03-29T11:20:26.224 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:20:26.224 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:26.224 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:20:26.224 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:20:26.224 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:26.224 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:20:26.225 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:20:26.225 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:20:26.225 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:20:26.225 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:20:26.225 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.244+0000 7fba5b243640 1 librados: starting msgr at 2024-03-29T11:20:26.225 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.244+0000 7fba5b243640 1 librados: starting objecter 2024-03-29T11:20:26.225 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.244+0000 7fba5b243640 1 Processor -- start 2024-03-29T11:20:26.225 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.244+0000 7fba5b243640 1 -- start start 2024-03-29T11:20:26.226 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.245+0000 7fba5b243640 1 librados: setting wanted keys 2024-03-29T11:20:26.226 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.245+0000 7fba5b243640 1 librados: calling monclient init 2024-03-29T11:20:26.226 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.245+0000 7fba5b243640 10 monclient: init 2024-03-29T11:20:26.226 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.246+0000 7fba5b243640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:26.226 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.246+0000 7fba5b243640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:20:26.226 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.248+0000 7fba5b243640 10 monclient(hunting): picked mon.noname-b con 0x7fba54071de0 addr v1:172.21.15.119:6789/0 2024-03-29T11:20:26.226 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.248+0000 7fba5b243640 10 monclient(hunting): picked mon.noname-a con 0x7fba541425d0 addr v1:172.21.15.104:6789/0 2024-03-29T11:20:26.226 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.248+0000 7fba5b243640 10 monclient(hunting): picked mon.noname-c con 0x7fba54131750 addr v1:172.21.15.104:6790/0 2024-03-29T11:20:26.227 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.248+0000 7fba5b243640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba540b3e70 con 0x7fba541425d0 2024-03-29T11:20:26.227 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.248+0000 7fba5b243640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba540b4150 con 0x7fba54071de0 2024-03-29T11:20:26.227 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.248+0000 7fba5b243640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba541d4de0 con 0x7fba54131750 2024-03-29T11:20:26.227 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.248+0000 7fba5b243640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:26.227 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.248+0000 7fba5b243640 10 monclient(hunting): authenticate will time out at 1007.499143s 2024-03-29T11:20:26.230 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.253+0000 7fba58fdf640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fba54071de0 0x7fba541d7c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:53169/0 (socket says 172.21.15.104:53169) 2024-03-29T11:20:26.230 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.253+0000 7fba58fdf640 1 -- 172.21.15.104:0/1123001168 learned_addr learned my addr 172.21.15.104:0/1123001168 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:20:26.230 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.253+0000 7fba53fff640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fba541425d0 0x7fba541d3fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59233/0 (socket says 172.21.15.104:59233) 2024-03-29T11:20:26.230 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.253+0000 7fba517fa640 10 client.?.objecter ms_handle_connect 0x7fba541425d0 2024-03-29T11:20:26.230 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.253+0000 7fba517fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:26.230 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.254+0000 7fba517fa640 1 -- 172.21.15.104:0/1123001168 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1212876158 0 0) 0x7fba540b3e70 con 0x7fba541425d0 2024-03-29T11:20:26.231 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.254+0000 7fba517fa640 10 monclient(hunting): _negotiate 2024-03-29T11:20:26.231 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.254+0000 7fba517fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:26.231 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.254+0000 7fba517fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:26.231 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.254+0000 7fba517fa640 10 monclient(hunting): my global_id is 4275 2024-03-29T11:20:26.231 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.254+0000 7fba517fa640 1 -- 172.21.15.104:0/1123001168 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba40004eb0 con 0x7fba541425d0 2024-03-29T11:20:26.231 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.254+0000 7fba517fa640 10 client.?.objecter ms_handle_connect 0x7fba54131750 2024-03-29T11:20:26.231 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.254+0000 7fba517fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:26.231 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.254+0000 7fba517fa640 1 -- 172.21.15.104:0/1123001168 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 165873592 0 0) 0x7fba541d4de0 con 0x7fba54131750 2024-03-29T11:20:26.231 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.254+0000 7fba517fa640 10 monclient(hunting): _negotiate 2024-03-29T11:20:26.231 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.254+0000 7fba517fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:26.231 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.254+0000 7fba517fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:26.231 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.254+0000 7fba517fa640 10 monclient(hunting): my global_id is 4277 2024-03-29T11:20:26.231 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.254+0000 7fba517fa640 1 -- 172.21.15.104:0/1123001168 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba540b3e70 con 0x7fba54131750 2024-03-29T11:20:26.231 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.254+0000 7fba517fa640 10 client.?.objecter ms_handle_connect 0x7fba54071de0 2024-03-29T11:20:26.231 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.254+0000 7fba517fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:26.233 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.254+0000 7fba517fa640 1 -- 172.21.15.104:0/1123001168 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2388140568 0 0) 0x7fba40004eb0 con 0x7fba541425d0 2024-03-29T11:20:26.233 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.254+0000 7fba517fa640 1 -- 172.21.15.104:0/1123001168 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba541d4de0 con 0x7fba541425d0 2024-03-29T11:20:26.233 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.255+0000 7fba517fa640 1 -- 172.21.15.104:0/1123001168 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fba48001630 con 0x7fba541425d0 2024-03-29T11:20:26.233 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.255+0000 7fba517fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:26.233 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.255+0000 7fba517fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:26.233 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.255+0000 7fba517fa640 10 monclient(hunting): dump: 2024-03-29T11:20:26.234 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:20:26.234 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:20:26.234 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:26.234 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:26.234 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:26.234 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:20:26.234 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:20:26.234 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:20:26.234 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:20:26.234 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:20:26.234 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.255+0000 7fba517fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:26.234 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.255+0000 7fba517fa640 1 -- 172.21.15.104:0/1123001168 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4286413522 0 0) 0x7fba540b4150 con 0x7fba54071de0 2024-03-29T11:20:26.234 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.255+0000 7fba517fa640 10 monclient(hunting): _negotiate 2024-03-29T11:20:26.234 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.255+0000 7fba517fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:26.235 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.255+0000 7fba517fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:26.235 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.255+0000 7fba517fa640 10 monclient(hunting): my global_id is 4255 2024-03-29T11:20:26.235 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.255+0000 7fba517fa640 1 -- 172.21.15.104:0/1123001168 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba40004eb0 con 0x7fba54071de0 2024-03-29T11:20:26.235 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.255+0000 7fba517fa640 1 -- 172.21.15.104:0/1123001168 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3444533768 0 0) 0x7fba540b3e70 con 0x7fba54131750 2024-03-29T11:20:26.235 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.255+0000 7fba517fa640 1 -- 172.21.15.104:0/1123001168 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba540b4150 con 0x7fba54131750 2024-03-29T11:20:26.235 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.255+0000 7fba517fa640 1 -- 172.21.15.104:0/1123001168 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fba4c001a10 con 0x7fba54131750 2024-03-29T11:20:26.235 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.255+0000 7fba517fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:26.235 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.255+0000 7fba517fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:26.235 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.255+0000 7fba517fa640 10 monclient(hunting): dump: 2024-03-29T11:20:26.235 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:20:26.235 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:20:26.235 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:26.235 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:26.235 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:26.235 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:20:26.235 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:20:26.236 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:20:26.236 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:20:26.236 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:20:26.236 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.255+0000 7fba517fa640 1 -- 172.21.15.104:0/1123001168 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3512506650 0 0) 0x7fba40004eb0 con 0x7fba54071de0 2024-03-29T11:20:26.236 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.255+0000 7fba517fa640 1 -- 172.21.15.104:0/1123001168 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba540b3e70 con 0x7fba54071de0 2024-03-29T11:20:26.236 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.255+0000 7fba517fa640 1 -- 172.21.15.104:0/1123001168 <== mon.0 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2122988994 0 0) 0x7fba541d4de0 con 0x7fba541425d0 2024-03-29T11:20:26.236 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.255+0000 7fba517fa640 1 -- 172.21.15.104:0/1123001168 >> v1:172.21.15.104:6790/0 conn(0x7fba54131750 legacy=0x7fba541d46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:26.236 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.255+0000 7fba517fa640 1 -- 172.21.15.104:0/1123001168 >> v1:172.21.15.119:6789/0 conn(0x7fba54071de0 legacy=0x7fba541d7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:26.236 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.255+0000 7fba517fa640 10 monclient: _finish_hunting 0 2024-03-29T11:20:26.236 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.255+0000 7fba517fa640 1 monclient: found mon.a 2024-03-29T11:20:26.236 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.256+0000 7fba517fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:26.236 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.256+0000 7fba517fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:20:26.236 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.256+0000 7fba517fa640 1 -- 172.21.15.104:0/1123001168 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba541d4f70 con 0x7fba541425d0 2024-03-29T11:20:26.236 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.256+0000 7fba517fa640 10 monclient: _finish_auth 0 2024-03-29T11:20:26.236 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.256+0000 7fba517fa640 10 monclient: _check_auth_tickets 2024-03-29T11:20:26.236 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.256+0000 7fba517fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:26.237 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.256+0000 7fba517fa640 1 -- 172.21.15.104:0/1123001168 <== mon.0 v1:172.21.15.104:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba48001790 con 0x7fba541425d0 2024-03-29T11:20:26.237 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.256+0000 7fba517fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:26.237 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.256+0000 7fba517fa640 1 -- 172.21.15.104:0/1123001168 <== mon.0 v1:172.21.15.104:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fba48004dd0 con 0x7fba541425d0 2024-03-29T11:20:26.237 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.256+0000 7fba517fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:26.237 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.256+0000 7fba517fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:26.237 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.256+0000 7fba517fa640 10 monclient: dump: 2024-03-29T11:20:26.237 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:20:26.237 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:20:26.237 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:26.237 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:26.237 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:26.237 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:20:26.237 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:20:26.237 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:20:26.237 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:20:26.237 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:20:26.238 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.256+0000 7fba527fc640 4 set_mon_vals no callback set 2024-03-29T11:20:26.238 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.256+0000 7fba5b243640 5 monclient: authenticate success, global_id 4275 2024-03-29T11:20:26.238 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.256+0000 7fba5b243640 10 monclient: _renew_subs 2024-03-29T11:20:26.238 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.256+0000 7fba5b243640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:20:26.238 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.256+0000 7fba5b243640 1 -- 172.21.15.104:0/1123001168 --> v1:172.21.15.104:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba541d7650 con 0x7fba541425d0 2024-03-29T11:20:26.238 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.256+0000 7fba5b243640 10 client.4275.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:26.238 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.256+0000 7fba5b243640 10 monclient: _renew_subs 2024-03-29T11:20:26.238 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.256+0000 7fba5b243640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:20:26.238 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.256+0000 7fba5b243640 1 -- 172.21.15.104:0/1123001168 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fba54132160 con 0x7fba541425d0 2024-03-29T11:20:26.238 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.256+0000 7fba5b243640 1 librados: init done 2024-03-29T11:20:26.240 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.259+0000 7fba517fa640 1 -- 172.21.15.104:0/1123001168 <== mon.0 v1:172.21.15.104:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7fba4801ada0 con 0x7fba541425d0 2024-03-29T11:20:26.240 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.262+0000 7fba5b243640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:26.240 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.262+0000 7fba5b243640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:26.240 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.262+0000 7fba5b243640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:20:26.240 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.262+0000 7fba5b243640 1 -- 172.21.15.104:0/1123001168 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba540aa9e0 con 0x7fba541425d0 2024-03-29T11:20:26.240 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.263+0000 7fba517fa640 1 -- 172.21.15.104:0/1123001168 <== mon.0 v1:172.21.15.104:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7fba48084ac0 con 0x7fba541425d0 2024-03-29T11:20:26.240 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.263+0000 7fba517fa640 10 client.4275.objecter ms_dispatch 0x7fba54000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:26.240 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.263+0000 7fba517fa640 3 client.4275.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:26.240 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.263+0000 7fba517fa640 3 client.4275.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:26.240 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.263+0000 7fba517fa640 20 client.4275.objecter dump_active .. 0 homeless 2024-03-29T11:20:26.240 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.263+0000 7fba58fdf640 10 monclient: get_auth_request con 0x7fba4006e1a0 auth_method 0 2024-03-29T11:20:26.240 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.263+0000 7fba517fa640 10 client.4275.objecter ms_handle_connect 0x7fba4006e1a0 2024-03-29T11:20:26.248 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.271+0000 7fba517fa640 1 -- 172.21.15.104:0/1123001168 <== mon.0 v1:172.21.15.104:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fba480503a0 con 0x7fba541425d0 2024-03-29T11:20:26.248 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.271+0000 7fba517fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:26.248 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.271+0000 7fba517fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:26.570 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.594+0000 7fba5b243640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:26.571 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.594+0000 7fba5b243640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:26.571 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.594+0000 7fba5b243640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:20:26.571 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.594+0000 7fba5b243640 1 -- 172.21.15.104:0/1123001168 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fba54075f10 con 0x7fba541425d0 2024-03-29T11:20:26.572 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.596+0000 7fba517fa640 1 -- 172.21.15.104:0/1123001168 <== mon.0 v1:172.21.15.104:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7fba480522d0 con 0x7fba541425d0 2024-03-29T11:20:26.572 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.596+0000 7fba517fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:26.573 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.596+0000 7fba517fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:26.574 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:20:26.574 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:26.577 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.601+0000 7fba5b243640 10 librados: watch_flush enter 2024-03-29T11:20:26.578 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.601+0000 7fba5b243640 10 librados: watch_flush exit 2024-03-29T11:20:26.578 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.601+0000 7fba5b243640 20 client.4275.objecter shutdown clearing up homeless session... 2024-03-29T11:20:26.578 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.601+0000 7fba5b243640 10 client.4275.objecter successfully canceled tick 2024-03-29T11:20:26.579 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.602+0000 7fba5b243640 1 -- 172.21.15.104:0/1123001168 >> v1:172.21.15.104:6816/3341126660 conn(0x7fba4006e1a0 legacy=0x7fba40070660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:26.579 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.602+0000 7fba5b243640 10 monclient: shutdown 2024-03-29T11:20:26.579 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.602+0000 7fba5b243640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:26.579 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.602+0000 7fba5b243640 1 -- 172.21.15.104:0/1123001168 >> v1:172.21.15.104:6789/0 conn(0x7fba541425d0 legacy=0x7fba541d3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:26.579 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.602+0000 7fba5b243640 1 -- 172.21.15.104:0/1123001168 shutdown_connections 2024-03-29T11:20:26.579 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.602+0000 7fba5b243640 1 -- 172.21.15.104:0/1123001168 >> 172.21.15.104:0/1123001168 conn(0x7fba5406b660 msgr2=0x7fba540a9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:26.579 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.602+0000 7fba5b243640 1 -- 172.21.15.104:0/1123001168 shutdown_connections 2024-03-29T11:20:26.579 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.602+0000 7fba5b243640 1 -- 172.21.15.104:0/1123001168 wait complete. 2024-03-29T11:20:26.579 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:26.602+0000 7fba5b243640 1 librados: shutdown 2024-03-29T11:20:26.596 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:33.403 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:33.561 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.585+0000 7f48589fb640 10 monclient: get_monmap_and_config 2024-03-29T11:20:33.562 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.585+0000 7f48589fb640 10 monclient: build_initial_monmap 2024-03-29T11:20:33.562 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.585+0000 7f48589fb640 10 monclient: monmap: 2024-03-29T11:20:33.562 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:20:33.562 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:20:33.562 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:20:33.586019+0000 2024-03-29T11:20:33.562 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:20:33.586019+0000 2024-03-29T11:20:33.562 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:33.563 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:20:33.563 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:20:33.563 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:20:33.563 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:20:33.563 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:20:33.567 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.586+0000 7f48589fb640 1 Processor -- start 2024-03-29T11:20:33.567 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.586+0000 7f48589fb640 1 -- start start 2024-03-29T11:20:33.567 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.586+0000 7f48589fb640 10 monclient: init 2024-03-29T11:20:33.567 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.587+0000 7f48589fb640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:33.568 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.587+0000 7f48589fb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:20:33.568 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.587+0000 7f48589fb640 10 monclient(hunting): picked mon.noname-a con 0x7f48541457f0 addr v1:172.21.15.104:6789/0 2024-03-29T11:20:33.568 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.587+0000 7f48589fb640 10 monclient(hunting): picked mon.noname-b con 0x7f485413f650 addr v1:172.21.15.104:6790/0 2024-03-29T11:20:33.568 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.587+0000 7f48589fb640 10 monclient(hunting): picked mon.noname-c con 0x7f48540a9d70 addr v1:172.21.15.119:6789/0 2024-03-29T11:20:33.568 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.587+0000 7f48589fb640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48540aaef0 con 0x7f48541457f0 2024-03-29T11:20:33.568 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.587+0000 7f48589fb640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4854145bd0 con 0x7f48540a9d70 2024-03-29T11:20:33.568 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.587+0000 7f48589fb640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4854145eb0 con 0x7f485413f650 2024-03-29T11:20:33.569 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.587+0000 7f48589fb640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:33.569 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.587+0000 7f48589fb640 10 monclient(hunting): authenticate will time out at 1014.838236s 2024-03-29T11:20:33.569 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.587+0000 7f4852d9d640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f48540a9d70 0x7f485414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:56232/0 (socket says 172.21.15.104:56232) 2024-03-29T11:20:33.569 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.587+0000 7f4852d9d640 1 -- 172.21.15.104:0/2328865275 learned_addr learned my addr 172.21.15.104:0/2328865275 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:20:33.569 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.588+0000 7f4851d9b640 1 -- 172.21.15.104:0/2328865275 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2762347178 0 0) 0x7f4854145eb0 con 0x7f485413f650 2024-03-29T11:20:33.569 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.588+0000 7f4851d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:20:33.569 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.588+0000 7f4851d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:33.570 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.588+0000 7f4851d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:33.570 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.588+0000 7f4851d9b640 10 monclient(hunting): my global_id is 4280 2024-03-29T11:20:33.570 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.588+0000 7f4851d9b640 1 -- 172.21.15.104:0/2328865275 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4840003520 con 0x7f485413f650 2024-03-29T11:20:33.570 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.588+0000 7f4851d9b640 1 -- 172.21.15.104:0/2328865275 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1851327467 0 0) 0x7f4854145bd0 con 0x7f48540a9d70 2024-03-29T11:20:33.570 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.588+0000 7f4851d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:20:33.570 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.588+0000 7f4851d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:33.570 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.588+0000 7f4851d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:33.571 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.588+0000 7f4851d9b640 10 monclient(hunting): my global_id is 4258 2024-03-29T11:20:33.571 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.588+0000 7f4851d9b640 1 -- 172.21.15.104:0/2328865275 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4854145eb0 con 0x7f48540a9d70 2024-03-29T11:20:33.571 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.588+0000 7f4851d9b640 1 -- 172.21.15.104:0/2328865275 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3855368470 0 0) 0x7f48540aaef0 con 0x7f48541457f0 2024-03-29T11:20:33.571 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.588+0000 7f4851d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:20:33.571 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.588+0000 7f4851d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:33.571 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.588+0000 7f4851d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:33.571 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.588+0000 7f4851d9b640 10 monclient(hunting): my global_id is 4278 2024-03-29T11:20:33.572 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.588+0000 7f4851d9b640 1 -- 172.21.15.104:0/2328865275 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4854145bd0 con 0x7f48541457f0 2024-03-29T11:20:33.572 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.588+0000 7f4851d9b640 1 -- 172.21.15.104:0/2328865275 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1243525016 0 0) 0x7f4854145eb0 con 0x7f48540a9d70 2024-03-29T11:20:33.572 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.588+0000 7f4851d9b640 1 -- 172.21.15.104:0/2328865275 >> v1:172.21.15.104:6790/0 conn(0x7f485413f650 legacy=0x7f48540b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:33.572 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.588+0000 7f4851d9b640 1 -- 172.21.15.104:0/2328865275 >> v1:172.21.15.104:6789/0 conn(0x7f48541457f0 legacy=0x7f485413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:33.572 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.588+0000 7f4851d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:20:33.572 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.588+0000 7f4851d9b640 1 monclient: found mon.noname-c 2024-03-29T11:20:33.572 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.588+0000 7f4851d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:33.572 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.588+0000 7f4851d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.119:6789/0 2024-03-29T11:20:33.573 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.588+0000 7f4851d9b640 1 -- 172.21.15.104:0/2328865275 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4854146340 con 0x7f48540a9d70 2024-03-29T11:20:33.573 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.588+0000 7f4851d9b640 10 monclient: _finish_auth 0 2024-03-29T11:20:33.573 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.588+0000 7f4851d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:20:33.573 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.588+0000 7f4851d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:33.573 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.588+0000 7f48589fb640 5 monclient: authenticate success, global_id 4258 2024-03-29T11:20:33.573 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.588+0000 7f48589fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:33.573 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.588+0000 7f4851d9b640 1 -- 172.21.15.104:0/2328865275 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f484c0019f0 con 0x7f48540a9d70 2024-03-29T11:20:33.573 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.588+0000 7f4851d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:33.574 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.588+0000 7f4851d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:33.574 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.588+0000 7f4851d9b640 10 monclient: dump: 2024-03-29T11:20:33.574 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:20:33.574 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:20:33.574 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:33.574 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:33.574 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:33.574 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:20:33.574 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:20:33.575 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:20:33.575 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:20:33.575 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:20:33.575 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.588+0000 7f48589fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:33.575 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.588+0000 7f4851d9b640 1 -- 172.21.15.104:0/2328865275 <== mon.1 v1:172.21.15.119:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f484c001c10 con 0x7f48540a9d70 2024-03-29T11:20:33.575 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.588+0000 7f4851d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:33.575 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.588+0000 7f48589fb640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:33.575 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.588+0000 7f48589fb640 4 set_mon_vals no callback set 2024-03-29T11:20:33.576 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.588+0000 7f4851d9b640 1 -- 172.21.15.104:0/2328865275 <== mon.1 v1:172.21.15.119:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f484c001fa0 con 0x7f48540a9d70 2024-03-29T11:20:33.576 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.588+0000 7f4851d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:33.576 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.588+0000 7f4851d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:33.576 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.588+0000 7f4851d9b640 10 monclient: dump: 2024-03-29T11:20:33.576 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:20:33.576 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:20:33.576 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:33.576 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:33.577 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:33.577 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:20:33.577 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:20:33.577 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:20:33.577 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:20:33.577 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:20:33.577 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.589+0000 7f48589fb640 10 monclient: shutdown 2024-03-29T11:20:33.577 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.589+0000 7f48589fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:33.578 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.589+0000 7f48589fb640 1 -- 172.21.15.104:0/2328865275 >> v1:172.21.15.119:6789/0 conn(0x7f48540a9d70 legacy=0x7f485414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:33.578 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.589+0000 7f48589fb640 1 -- 172.21.15.104:0/2328865275 shutdown_connections 2024-03-29T11:20:33.578 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.589+0000 7f48589fb640 1 -- 172.21.15.104:0/2328865275 >> 172.21.15.104:0/2328865275 conn(0x7f4854066a00 msgr2=0x7f4854066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:33.578 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.590+0000 7f48589fb640 1 -- 172.21.15.104:0/2328865275 shutdown_connections 2024-03-29T11:20:33.578 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.590+0000 7f48589fb640 1 -- 172.21.15.104:0/2328865275 wait complete. 2024-03-29T11:20:33.578 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.591+0000 7f48589fb640 10 monclient: build_initial_monmap 2024-03-29T11:20:33.578 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.591+0000 7f48589fb640 10 monclient: monmap: 2024-03-29T11:20:33.578 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:20:33.579 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:33.579 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:20:33.579 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:20:33.579 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:33.579 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:20:33.579 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:20:33.579 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:20:33.579 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:20:33.579 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:20:33.587 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.591+0000 7f48589fb640 1 librados: starting msgr at 2024-03-29T11:20:33.588 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.591+0000 7f48589fb640 1 librados: starting objecter 2024-03-29T11:20:33.588 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.591+0000 7f48589fb640 1 Processor -- start 2024-03-29T11:20:33.588 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.592+0000 7f48589fb640 1 -- start start 2024-03-29T11:20:33.588 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.592+0000 7f48589fb640 1 librados: setting wanted keys 2024-03-29T11:20:33.588 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.592+0000 7f48589fb640 1 librados: calling monclient init 2024-03-29T11:20:33.588 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.592+0000 7f48589fb640 10 monclient: init 2024-03-29T11:20:33.589 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.592+0000 7f48589fb640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:33.589 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.592+0000 7f48589fb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:20:33.589 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.592+0000 7f48589fb640 10 monclient(hunting): picked mon.noname-b con 0x7f48540a9d70 addr v1:172.21.15.119:6789/0 2024-03-29T11:20:33.589 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.593+0000 7f48589fb640 10 monclient(hunting): picked mon.noname-c con 0x7f485413f650 addr v1:172.21.15.104:6790/0 2024-03-29T11:20:33.589 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.593+0000 7f485259c640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f48540a9d70 0x7f4854141aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:56720/0 (socket says 172.21.15.104:56720) 2024-03-29T11:20:33.590 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.593+0000 7f4849d9b640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f485413f650 0x7f48541deef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:60892/0 (socket says 172.21.15.104:60892) 2024-03-29T11:20:33.590 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.593+0000 7f485259c640 1 -- 172.21.15.104:0/4196159473 learned_addr learned my addr 172.21.15.104:0/4196159473 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:20:33.590 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.593+0000 7f48589fb640 10 monclient(hunting): picked mon.noname-a con 0x7f48541457f0 addr v1:172.21.15.104:6789/0 2024-03-29T11:20:33.590 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.593+0000 7f48589fb640 1 -- 172.21.15.104:0/4196159473 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f485406e9f0 con 0x7f48541457f0 2024-03-29T11:20:33.590 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.593+0000 7f48589fb640 1 -- 172.21.15.104:0/4196159473 --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48541dfd10 con 0x7f48540a9d70 2024-03-29T11:20:33.591 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.593+0000 7f484b7fe640 10 client.?.objecter ms_handle_connect 0x7f48540a9d70 2024-03-29T11:20:33.591 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.593+0000 7f484b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:33.591 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.593+0000 7f48589fb640 1 -- 172.21.15.104:0/4196159473 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f485406ecd0 con 0x7f485413f650 2024-03-29T11:20:33.591 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.593+0000 7f48589fb640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:33.591 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.593+0000 7f48589fb640 10 monclient(hunting): authenticate will time out at 1014.844625s 2024-03-29T11:20:33.592 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.593+0000 7f484b7fe640 10 client.?.objecter ms_handle_connect 0x7f485413f650 2024-03-29T11:20:33.592 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.593+0000 7f484b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:33.592 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.593+0000 7f484b7fe640 1 -- 172.21.15.104:0/4196159473 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1997527026 0 0) 0x7f485406ecd0 con 0x7f485413f650 2024-03-29T11:20:33.592 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.593+0000 7f484b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:33.592 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.593+0000 7f484b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:33.592 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.593+0000 7f484b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:33.593 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.593+0000 7f484b7fe640 10 monclient(hunting): my global_id is 4283 2024-03-29T11:20:33.593 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.593+0000 7f484b7fe640 1 -- 172.21.15.104:0/4196159473 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4834004eb0 con 0x7f485413f650 2024-03-29T11:20:33.593 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.593+0000 7f484b7fe640 1 -- 172.21.15.104:0/4196159473 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2973857341 0 0) 0x7f48541dfd10 con 0x7f48540a9d70 2024-03-29T11:20:33.593 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.593+0000 7f484b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:33.593 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.593+0000 7f484b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:33.594 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.593+0000 7f484b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:33.594 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.593+0000 7f484b7fe640 10 monclient(hunting): my global_id is 4261 2024-03-29T11:20:33.594 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.594+0000 7f484b7fe640 1 -- 172.21.15.104:0/4196159473 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f485406ecd0 con 0x7f48540a9d70 2024-03-29T11:20:33.594 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.594+0000 7f484b7fe640 10 client.?.objecter ms_handle_connect 0x7f48541457f0 2024-03-29T11:20:33.594 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.594+0000 7f484b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:33.595 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.594+0000 7f484b7fe640 1 -- 172.21.15.104:0/4196159473 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1921045983 0 0) 0x7f485406ecd0 con 0x7f48540a9d70 2024-03-29T11:20:33.595 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.594+0000 7f484b7fe640 1 -- 172.21.15.104:0/4196159473 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48541dfd10 con 0x7f48540a9d70 2024-03-29T11:20:33.595 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.594+0000 7f484b7fe640 1 -- 172.21.15.104:0/4196159473 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f483c0018d0 con 0x7f48540a9d70 2024-03-29T11:20:33.595 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.594+0000 7f484b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:33.595 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.594+0000 7f484b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:33.595 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.594+0000 7f484b7fe640 10 monclient(hunting): dump: 2024-03-29T11:20:33.596 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:20:33.596 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:20:33.596 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:33.596 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:33.596 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:33.596 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:20:33.597 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:20:33.597 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:20:33.597 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:20:33.597 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:20:33.597 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.594+0000 7f484b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:33.598 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.594+0000 7f484b7fe640 1 -- 172.21.15.104:0/4196159473 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1574407637 0 0) 0x7f485406e9f0 con 0x7f48541457f0 2024-03-29T11:20:33.598 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.594+0000 7f484b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:33.598 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.594+0000 7f484b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:33.598 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.594+0000 7f484b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:33.598 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.594+0000 7f484b7fe640 10 monclient(hunting): my global_id is 4281 2024-03-29T11:20:33.598 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.594+0000 7f484b7fe640 1 -- 172.21.15.104:0/4196159473 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f485406ecd0 con 0x7f48541457f0 2024-03-29T11:20:33.599 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.594+0000 7f484b7fe640 1 -- 172.21.15.104:0/4196159473 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3006461711 0 0) 0x7f4834004eb0 con 0x7f485413f650 2024-03-29T11:20:33.599 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.594+0000 7f484b7fe640 1 -- 172.21.15.104:0/4196159473 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f485406e9f0 con 0x7f485413f650 2024-03-29T11:20:33.599 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.594+0000 7f484b7fe640 1 -- 172.21.15.104:0/4196159473 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f48440018d0 con 0x7f485413f650 2024-03-29T11:20:33.599 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.594+0000 7f484b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:33.599 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.595+0000 7f484b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:33.600 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.595+0000 7f484b7fe640 10 monclient(hunting): dump: 2024-03-29T11:20:33.600 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:20:33.600 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:20:33.600 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:33.600 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:33.600 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:33.601 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:20:33.601 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:20:33.601 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:20:33.601 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:20:33.601 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:20:33.601 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.595+0000 7f484b7fe640 1 -- 172.21.15.104:0/4196159473 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2605211555 0 0) 0x7f48541dfd10 con 0x7f48540a9d70 2024-03-29T11:20:33.602 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.595+0000 7f484b7fe640 1 -- 172.21.15.104:0/4196159473 >> v1:172.21.15.104:6790/0 conn(0x7f485413f650 legacy=0x7f48541deef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:33.602 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.595+0000 7f484b7fe640 1 -- 172.21.15.104:0/4196159473 >> v1:172.21.15.104:6789/0 conn(0x7f48541457f0 legacy=0x7f48541df600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:33.602 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.595+0000 7f484b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:20:33.602 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.595+0000 7f484b7fe640 1 monclient: found mon.b 2024-03-29T11:20:33.602 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.595+0000 7f484b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:33.603 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.595+0000 7f484b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:20:33.603 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.595+0000 7f484b7fe640 1 -- 172.21.15.104:0/4196159473 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f485406efb0 con 0x7f48540a9d70 2024-03-29T11:20:33.603 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.595+0000 7f484b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:20:33.603 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.595+0000 7f484b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:20:33.603 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.595+0000 7f484b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:33.603 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.595+0000 7f48589fb640 5 monclient: authenticate success, global_id 4261 2024-03-29T11:20:33.604 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.595+0000 7f48589fb640 10 monclient: _renew_subs 2024-03-29T11:20:33.604 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.595+0000 7f48589fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:20:33.604 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.595+0000 7f48589fb640 1 -- 172.21.15.104:0/4196159473 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4854142770 con 0x7f48540a9d70 2024-03-29T11:20:33.604 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.596+0000 7f48589fb640 10 client.4261.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:33.604 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.596+0000 7f48589fb640 10 monclient: _renew_subs 2024-03-29T11:20:33.605 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.596+0000 7f48589fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:20:33.605 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.596+0000 7f48589fb640 1 -- 172.21.15.104:0/4196159473 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f48540720b0 con 0x7f48540a9d70 2024-03-29T11:20:33.605 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.596+0000 7f48589fb640 1 librados: init done 2024-03-29T11:20:33.605 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.596+0000 7f484b7fe640 1 -- 172.21.15.104:0/4196159473 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f483c0035f0 con 0x7f48540a9d70 2024-03-29T11:20:33.605 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.596+0000 7f484b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:33.605 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.596+0000 7f484b7fe640 1 -- 172.21.15.104:0/4196159473 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f483c0037a0 con 0x7f48540a9d70 2024-03-29T11:20:33.606 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.596+0000 7f484b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:33.606 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.596+0000 7f484b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:33.606 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.596+0000 7f484b7fe640 10 monclient: dump: 2024-03-29T11:20:33.606 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:20:33.606 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:20:33.606 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:33.607 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:33.607 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:33.607 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:20:33.607 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:20:33.607 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:20:33.607 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:20:33.608 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:20:33.608 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.596+0000 7f485159a640 4 set_mon_vals no callback set 2024-03-29T11:20:33.608 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.598+0000 7f48589fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:33.608 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.598+0000 7f48589fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:33.608 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.598+0000 7f48589fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:20:33.608 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.598+0000 7f484b7fe640 1 -- 172.21.15.104:0/4196159473 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f483c003900 con 0x7f48540a9d70 2024-03-29T11:20:33.609 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.598+0000 7f48589fb640 1 -- 172.21.15.104:0/4196159473 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4838005080 con 0x7f48540a9d70 2024-03-29T11:20:33.609 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.599+0000 7f4849d9b640 10 monclient: get_auth_request con 0x7f483406e000 auth_method 0 2024-03-29T11:20:33.609 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.600+0000 7f484b7fe640 10 client.4261.objecter ms_handle_connect 0x7f483406e000 2024-03-29T11:20:33.609 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.600+0000 7f484b7fe640 1 -- 172.21.15.104:0/4196159473 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f483c084810 con 0x7f48540a9d70 2024-03-29T11:20:33.609 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.600+0000 7f484b7fe640 10 client.4261.objecter ms_dispatch 0x7f4854000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:33.610 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.600+0000 7f484b7fe640 3 client.4261.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:33.610 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.600+0000 7f484b7fe640 3 client.4261.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:33.610 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.600+0000 7f484b7fe640 20 client.4261.objecter dump_active .. 0 homeless 2024-03-29T11:20:33.610 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.607+0000 7f484b7fe640 1 -- 172.21.15.104:0/4196159473 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f483c0500f0 con 0x7f48540a9d70 2024-03-29T11:20:33.610 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.607+0000 7f484b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:33.611 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.607+0000 7f484b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:33.975 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.999+0000 7f48589fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:33.975 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.999+0000 7f48589fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:33.975 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.999+0000 7f48589fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:20:33.976 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:33.999+0000 7f48589fb640 1 -- 172.21.15.104:0/4196159473 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4838005330 con 0x7f48540a9d70 2024-03-29T11:20:33.976 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:34.000+0000 7f484b7fe640 1 -- 172.21.15.104:0/4196159473 <== mon.1 v1:172.21.15.119:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f483c052020 con 0x7f48540a9d70 2024-03-29T11:20:33.976 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:34.000+0000 7f484b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:33.976 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:34.000+0000 7f484b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:33.976 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:20:33.977 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:33.980 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:34.003+0000 7f48589fb640 10 librados: watch_flush enter 2024-03-29T11:20:33.980 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:34.003+0000 7f48589fb640 10 librados: watch_flush exit 2024-03-29T11:20:33.980 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:34.003+0000 7f48589fb640 20 client.4261.objecter shutdown clearing up homeless session... 2024-03-29T11:20:33.980 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:34.003+0000 7f48589fb640 10 client.4261.objecter successfully canceled tick 2024-03-29T11:20:33.980 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:34.003+0000 7f48589fb640 1 -- 172.21.15.104:0/4196159473 >> v1:172.21.15.104:6816/3341126660 conn(0x7f483406e000 legacy=0x7f48340704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:33.980 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:34.003+0000 7f48589fb640 10 monclient: shutdown 2024-03-29T11:20:33.980 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:34.003+0000 7f48589fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:33.981 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:34.004+0000 7f48589fb640 1 -- 172.21.15.104:0/4196159473 >> v1:172.21.15.119:6789/0 conn(0x7f48540a9d70 legacy=0x7f4854141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:33.981 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:34.004+0000 7f48589fb640 1 -- 172.21.15.104:0/4196159473 shutdown_connections 2024-03-29T11:20:33.981 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:34.004+0000 7f48589fb640 1 -- 172.21.15.104:0/4196159473 >> 172.21.15.104:0/4196159473 conn(0x7f4854066a00 msgr2=0x7f485413aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:33.981 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:34.004+0000 7f48589fb640 1 -- 172.21.15.104:0/4196159473 shutdown_connections 2024-03-29T11:20:33.981 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:34.004+0000 7f48589fb640 1 -- 172.21.15.104:0/4196159473 wait complete. 2024-03-29T11:20:33.981 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:34.004+0000 7f48589fb640 1 librados: shutdown 2024-03-29T11:20:33.997 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:40.804 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:40.967 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.990+0000 7fe109b15640 10 monclient: get_monmap_and_config 2024-03-29T11:20:40.967 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.990+0000 7fe109b15640 10 monclient: build_initial_monmap 2024-03-29T11:20:40.967 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.990+0000 7fe109b15640 10 monclient: monmap: 2024-03-29T11:20:40.967 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:20:40.967 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:20:40.967 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:20:40.991691+0000 2024-03-29T11:20:40.967 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:20:40.991691+0000 2024-03-29T11:20:40.967 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:40.967 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:20:40.968 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:20:40.968 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:20:40.968 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:20:40.968 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:20:40.968 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.992+0000 7fe109b15640 1 Processor -- start 2024-03-29T11:20:40.968 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.992+0000 7fe109b15640 1 -- start start 2024-03-29T11:20:40.968 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.992+0000 7fe109b15640 10 monclient: init 2024-03-29T11:20:40.970 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.994+0000 7fe109b15640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:40.971 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.994+0000 7fe109b15640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:20:40.971 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.994+0000 7fe109b15640 10 monclient(hunting): picked mon.noname-a con 0x7fe1040a9750 addr v1:172.21.15.104:6789/0 2024-03-29T11:20:40.971 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.994+0000 7fe109b15640 10 monclient(hunting): picked mon.noname-b con 0x7fe1040b04e0 addr v1:172.21.15.104:6790/0 2024-03-29T11:20:40.971 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.994+0000 7fe109b15640 10 monclient(hunting): picked mon.noname-c con 0x7fe1040af800 addr v1:172.21.15.119:6789/0 2024-03-29T11:20:40.971 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.994+0000 7fe109b15640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1040ab380 con 0x7fe1040a9750 2024-03-29T11:20:40.971 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.994+0000 7fe109b15640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1040ab660 con 0x7fe1040af800 2024-03-29T11:20:40.971 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.994+0000 7fe109b15640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1040abd20 con 0x7fe1040b04e0 2024-03-29T11:20:40.971 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.994+0000 7fe109b15640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:40.971 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.994+0000 7fe109b15640 10 monclient(hunting): authenticate will time out at 1022.245201s 2024-03-29T11:20:40.971 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.994+0000 7fe1037fe640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fe1040a9750 0x7fe1040ac140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:44056/0 (socket says 172.21.15.104:44056) 2024-03-29T11:20:40.971 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.994+0000 7fe1037fe640 1 -- 172.21.15.104:0/4273515855 learned_addr learned my addr 172.21.15.104:0/4273515855 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:20:40.972 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.995+0000 7fe1027fc640 1 -- 172.21.15.104:0/4273515855 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 385781256 0 0) 0x7fe1040ab660 con 0x7fe1040af800 2024-03-29T11:20:40.972 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.995+0000 7fe1027fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:40.972 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.995+0000 7fe1027fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:40.972 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.995+0000 7fe1027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:40.972 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.995+0000 7fe1027fc640 10 monclient(hunting): my global_id is 4264 2024-03-29T11:20:40.972 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.995+0000 7fe1027fc640 1 -- 172.21.15.104:0/4273515855 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0ec003520 con 0x7fe1040af800 2024-03-29T11:20:40.972 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.995+0000 7fe1027fc640 1 -- 172.21.15.104:0/4273515855 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3819139060 0 0) 0x7fe1040ab380 con 0x7fe1040a9750 2024-03-29T11:20:40.972 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.995+0000 7fe1027fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:40.973 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.995+0000 7fe1027fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:40.973 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.995+0000 7fe1027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:40.973 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.995+0000 7fe1027fc640 10 monclient(hunting): my global_id is 4284 2024-03-29T11:20:40.973 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.995+0000 7fe1027fc640 1 -- 172.21.15.104:0/4273515855 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1040ab660 con 0x7fe1040a9750 2024-03-29T11:20:40.974 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.996+0000 7fe1027fc640 1 -- 172.21.15.104:0/4273515855 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4039315528 0 0) 0x7fe0ec003520 con 0x7fe1040af800 2024-03-29T11:20:40.974 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.996+0000 7fe1027fc640 1 -- 172.21.15.104:0/4273515855 >> v1:172.21.15.104:6790/0 conn(0x7fe1040b04e0 legacy=0x7fe1040af0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:40.974 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.996+0000 7fe1027fc640 1 -- 172.21.15.104:0/4273515855 >> v1:172.21.15.104:6789/0 conn(0x7fe1040a9750 legacy=0x7fe1040ac140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:40.974 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.996+0000 7fe1027fc640 10 monclient: _finish_hunting 0 2024-03-29T11:20:40.974 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.996+0000 7fe1027fc640 1 monclient: found mon.noname-c 2024-03-29T11:20:40.974 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.996+0000 7fe1027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:40.975 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.996+0000 7fe1027fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.119:6789/0 2024-03-29T11:20:40.975 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.996+0000 7fe1027fc640 1 -- 172.21.15.104:0/4273515855 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1040aa520 con 0x7fe1040af800 2024-03-29T11:20:40.975 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.996+0000 7fe1027fc640 10 monclient: _finish_auth 0 2024-03-29T11:20:40.975 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.996+0000 7fe1027fc640 10 monclient: _check_auth_tickets 2024-03-29T11:20:40.975 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.996+0000 7fe1027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:40.975 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.996+0000 7fe109b15640 5 monclient: authenticate success, global_id 4264 2024-03-29T11:20:40.975 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.996+0000 7fe109b15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:40.975 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.996+0000 7fe1027fc640 1 -- 172.21.15.104:0/4273515855 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fe0fc001a10 con 0x7fe1040af800 2024-03-29T11:20:40.975 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.996+0000 7fe1027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:40.975 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.996+0000 7fe1027fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:40.975 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.996+0000 7fe1027fc640 10 monclient: dump: 2024-03-29T11:20:40.975 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:20:40.975 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:20:40.976 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:40.976 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:40.976 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:40.976 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:20:40.976 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:20:40.976 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:20:40.976 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:20:40.976 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:20:40.976 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.996+0000 7fe109b15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:40.976 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.996+0000 7fe1027fc640 1 -- 172.21.15.104:0/4273515855 <== mon.1 v1:172.21.15.119:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0fc001b70 con 0x7fe1040af800 2024-03-29T11:20:40.976 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.996+0000 7fe1027fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:40.976 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.996+0000 7fe1027fc640 1 -- 172.21.15.104:0/4273515855 <== mon.1 v1:172.21.15.119:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fe0fc001f80 con 0x7fe1040af800 2024-03-29T11:20:40.976 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.996+0000 7fe1027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:40.977 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.996+0000 7fe1027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:40.977 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.996+0000 7fe1027fc640 10 monclient: dump: 2024-03-29T11:20:40.977 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:20:40.977 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:20:40.977 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:40.977 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:40.977 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:40.977 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:20:40.977 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:20:40.977 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:20:40.977 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:20:40.977 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:20:40.977 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.996+0000 7fe109b15640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:40.977 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.996+0000 7fe109b15640 4 set_mon_vals no callback set 2024-03-29T11:20:40.977 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.997+0000 7fe109b15640 10 monclient: shutdown 2024-03-29T11:20:40.978 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.997+0000 7fe109b15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:40.978 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.997+0000 7fe109b15640 1 -- 172.21.15.104:0/4273515855 >> v1:172.21.15.119:6789/0 conn(0x7fe1040af800 legacy=0x7fe10414aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:40.978 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.997+0000 7fe109b15640 1 -- 172.21.15.104:0/4273515855 shutdown_connections 2024-03-29T11:20:40.978 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.997+0000 7fe109b15640 1 -- 172.21.15.104:0/4273515855 >> 172.21.15.104:0/4273515855 conn(0x7fe1040672a0 msgr2=0x7fe1040b46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:40.978 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.998+0000 7fe109b15640 1 -- 172.21.15.104:0/4273515855 shutdown_connections 2024-03-29T11:20:40.988 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.998+0000 7fe109b15640 1 -- 172.21.15.104:0/4273515855 wait complete. 2024-03-29T11:20:40.989 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.999+0000 7fe109b15640 10 monclient: build_initial_monmap 2024-03-29T11:20:40.989 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.999+0000 7fe109b15640 10 monclient: monmap: 2024-03-29T11:20:40.989 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:20:40.989 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:40.989 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:20:40.989 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:20:40.989 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:40.989 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:20:40.989 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:20:40.989 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:20:40.989 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:20:40.989 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:20:40.990 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.999+0000 7fe109b15640 1 librados: starting msgr at 2024-03-29T11:20:40.990 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.999+0000 7fe109b15640 1 librados: starting objecter 2024-03-29T11:20:40.990 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:40.999+0000 7fe109b15640 1 Processor -- start 2024-03-29T11:20:40.990 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.000+0000 7fe109b15640 1 -- start start 2024-03-29T11:20:40.990 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.000+0000 7fe109b15640 1 librados: setting wanted keys 2024-03-29T11:20:40.990 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.000+0000 7fe109b15640 1 librados: calling monclient init 2024-03-29T11:20:40.990 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.000+0000 7fe109b15640 10 monclient: init 2024-03-29T11:20:40.990 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.001+0000 7fe109b15640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:40.990 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.001+0000 7fe109b15640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:20:40.990 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.001+0000 7fe109b15640 10 monclient(hunting): picked mon.noname-c con 0x7fe1040a9750 addr v1:172.21.15.104:6790/0 2024-03-29T11:20:40.990 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.001+0000 7fe109b15640 10 monclient(hunting): picked mon.noname-a con 0x7fe1040af800 addr v1:172.21.15.104:6789/0 2024-03-29T11:20:40.990 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.001+0000 7fe109b15640 10 monclient(hunting): picked mon.noname-b con 0x7fe1040b04e0 addr v1:172.21.15.119:6789/0 2024-03-29T11:20:40.990 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.001+0000 7fe109b15640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1041a4390 con 0x7fe1040af800 2024-03-29T11:20:40.991 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.002+0000 7fe109b15640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1041a1eb0 con 0x7fe1040b04e0 2024-03-29T11:20:40.991 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.002+0000 7fe109b15640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1041d70c0 con 0x7fe1040a9750 2024-03-29T11:20:40.991 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.002+0000 7fe109b15640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:40.991 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.002+0000 7fe1037fe640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7fe1040a9750 0x7fe1041d69b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:48378/0 (socket says 172.21.15.104:48378) 2024-03-29T11:20:40.991 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.002+0000 7fe1037fe640 1 -- 172.21.15.104:0/2697329516 learned_addr learned my addr 172.21.15.104:0/2697329516 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:20:40.991 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.002+0000 7fe109b15640 10 monclient(hunting): authenticate will time out at 1022.253526s 2024-03-29T11:20:40.991 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.002+0000 7fe0ebfff640 10 client.?.objecter ms_handle_connect 0x7fe1040a9750 2024-03-29T11:20:40.991 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.002+0000 7fe0ebfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:40.991 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.002+0000 7fe0ebfff640 10 client.?.objecter ms_handle_connect 0x7fe1040b04e0 2024-03-29T11:20:40.991 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.002+0000 7fe0ebfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:40.991 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.002+0000 7fe0ebfff640 1 -- 172.21.15.104:0/2697329516 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4097850512 0 0) 0x7fe1041d70c0 con 0x7fe1040a9750 2024-03-29T11:20:40.991 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.002+0000 7fe0ebfff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:40.991 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.003+0000 7fe0ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:40.991 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.003+0000 7fe0ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:40.992 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.003+0000 7fe0ebfff640 10 monclient(hunting): my global_id is 4289 2024-03-29T11:20:40.992 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.003+0000 7fe0ebfff640 1 -- 172.21.15.104:0/2697329516 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0f0004eb0 con 0x7fe1040a9750 2024-03-29T11:20:40.992 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.003+0000 7fe0ebfff640 10 client.?.objecter ms_handle_connect 0x7fe1040af800 2024-03-29T11:20:40.992 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.003+0000 7fe0ebfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:40.992 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.003+0000 7fe0ebfff640 1 -- 172.21.15.104:0/2697329516 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2022303932 0 0) 0x7fe1041a1eb0 con 0x7fe1040b04e0 2024-03-29T11:20:40.992 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.003+0000 7fe0ebfff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:40.992 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.003+0000 7fe0ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:40.992 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.003+0000 7fe0ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:40.992 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.003+0000 7fe0ebfff640 10 monclient(hunting): my global_id is 4267 2024-03-29T11:20:40.992 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.003+0000 7fe0ebfff640 1 -- 172.21.15.104:0/2697329516 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1041d70c0 con 0x7fe1040b04e0 2024-03-29T11:20:40.992 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.003+0000 7fe0ebfff640 1 -- 172.21.15.104:0/2697329516 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1807969625 0 0) 0x7fe1041a4390 con 0x7fe1040af800 2024-03-29T11:20:40.992 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.003+0000 7fe0ebfff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:40.992 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.003+0000 7fe0ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:40.992 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.003+0000 7fe0ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:40.993 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.003+0000 7fe0ebfff640 10 monclient(hunting): my global_id is 4287 2024-03-29T11:20:40.993 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.003+0000 7fe0ebfff640 1 -- 172.21.15.104:0/2697329516 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1041a1eb0 con 0x7fe1040af800 2024-03-29T11:20:40.993 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.004+0000 7fe0ebfff640 1 -- 172.21.15.104:0/2697329516 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3493226408 0 0) 0x7fe1041d70c0 con 0x7fe1040b04e0 2024-03-29T11:20:40.993 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.004+0000 7fe0ebfff640 1 -- 172.21.15.104:0/2697329516 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1041a4390 con 0x7fe1040b04e0 2024-03-29T11:20:40.993 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.004+0000 7fe0ebfff640 1 -- 172.21.15.104:0/2697329516 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fe0fc001f80 con 0x7fe1040b04e0 2024-03-29T11:20:40.993 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.004+0000 7fe0ebfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:40.993 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.004+0000 7fe0ebfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:40.993 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.004+0000 7fe0ebfff640 10 monclient(hunting): dump: 2024-03-29T11:20:40.993 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:20:40.993 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:20:40.993 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:40.993 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:40.993 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:40.993 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:20:40.994 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:20:40.994 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:20:40.994 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:20:40.994 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:20:40.994 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.004+0000 7fe0ebfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:40.994 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.004+0000 7fe0ebfff640 1 -- 172.21.15.104:0/2697329516 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2174294425 0 0) 0x7fe1041a4390 con 0x7fe1040b04e0 2024-03-29T11:20:40.994 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.004+0000 7fe0ebfff640 1 -- 172.21.15.104:0/2697329516 >> v1:172.21.15.104:6790/0 conn(0x7fe1040a9750 legacy=0x7fe1041d69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:40.994 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.004+0000 7fe0ebfff640 1 -- 172.21.15.104:0/2697329516 >> v1:172.21.15.104:6789/0 conn(0x7fe1040af800 legacy=0x7fe1041a23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:40.994 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.004+0000 7fe0ebfff640 10 monclient: _finish_hunting 0 2024-03-29T11:20:40.994 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.004+0000 7fe0ebfff640 1 monclient: found mon.b 2024-03-29T11:20:40.994 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.004+0000 7fe0ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:40.994 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.004+0000 7fe0ebfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:20:40.994 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.004+0000 7fe0ebfff640 1 -- 172.21.15.104:0/2697329516 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1041d7250 con 0x7fe1040b04e0 2024-03-29T11:20:40.995 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.004+0000 7fe0ebfff640 10 monclient: _finish_auth 0 2024-03-29T11:20:40.995 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.004+0000 7fe0ebfff640 10 monclient: _check_auth_tickets 2024-03-29T11:20:40.995 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.004+0000 7fe0ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:40.995 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.004+0000 7fe109b15640 5 monclient: authenticate success, global_id 4267 2024-03-29T11:20:40.995 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.004+0000 7fe109b15640 10 monclient: _renew_subs 2024-03-29T11:20:40.995 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.004+0000 7fe109b15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:20:40.995 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.004+0000 7fe109b15640 1 -- 172.21.15.104:0/2697329516 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1041d7500 con 0x7fe1040b04e0 2024-03-29T11:20:40.995 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.004+0000 7fe109b15640 10 client.4267.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:40.995 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.004+0000 7fe109b15640 10 monclient: _renew_subs 2024-03-29T11:20:40.995 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.004+0000 7fe109b15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:20:40.995 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.004+0000 7fe109b15640 1 -- 172.21.15.104:0/2697329516 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1041deeb0 con 0x7fe1040b04e0 2024-03-29T11:20:40.995 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.004+0000 7fe109b15640 1 librados: init done 2024-03-29T11:20:40.995 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.005+0000 7fe0ebfff640 1 -- 172.21.15.104:0/2697329516 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0fc0015b0 con 0x7fe1040b04e0 2024-03-29T11:20:40.996 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.005+0000 7fe0ebfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:40.996 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.005+0000 7fe0ebfff640 1 -- 172.21.15.104:0/2697329516 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fe0fc0019f0 con 0x7fe1040b04e0 2024-03-29T11:20:40.996 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.005+0000 7fe0ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:40.996 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.005+0000 7fe0ebfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:40.996 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.005+0000 7fe0ebfff640 10 monclient: dump: 2024-03-29T11:20:40.996 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:20:40.996 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:20:40.996 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:40.996 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:40.996 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:40.996 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:20:40.996 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:20:40.996 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:20:40.996 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:20:40.997 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:20:40.997 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.005+0000 7fe101ffb640 4 set_mon_vals no callback set 2024-03-29T11:20:40.997 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.006+0000 7fe109b15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:40.997 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.007+0000 7fe109b15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:40.997 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.007+0000 7fe109b15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:20:40.997 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.007+0000 7fe109b15640 1 -- 172.21.15.104:0/2697329516 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0e4005080 con 0x7fe1040b04e0 2024-03-29T11:20:40.997 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.007+0000 7fe0ebfff640 1 -- 172.21.15.104:0/2697329516 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7fe0fc019c30 con 0x7fe1040b04e0 2024-03-29T11:20:40.997 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.008+0000 7fe0ebfff640 1 -- 172.21.15.104:0/2697329516 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7fe0fc083c10 con 0x7fe1040b04e0 2024-03-29T11:20:40.997 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.008+0000 7fe0ebfff640 10 client.4267.objecter ms_dispatch 0x7fe104000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:40.997 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.008+0000 7fe0ebfff640 3 client.4267.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:40.997 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.008+0000 7fe0ebfff640 3 client.4267.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:40.997 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.008+0000 7fe0ebfff640 20 client.4267.objecter dump_active .. 0 homeless 2024-03-29T11:20:40.997 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.008+0000 7fe1037fe640 10 monclient: get_auth_request con 0x7fe0f006dbb0 auth_method 0 2024-03-29T11:20:40.997 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.008+0000 7fe0ebfff640 10 client.4267.objecter ms_handle_connect 0x7fe0f006dbb0 2024-03-29T11:20:40.998 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.015+0000 7fe0ebfff640 1 -- 172.21.15.104:0/2697329516 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe0fc0530b0 con 0x7fe1040b04e0 2024-03-29T11:20:40.998 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.015+0000 7fe0ebfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:40.998 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.015+0000 7fe0ebfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:41.410 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.434+0000 7fe109b15640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:41.411 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.434+0000 7fe109b15640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:41.411 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.434+0000 7fe109b15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:20:41.411 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.434+0000 7fe109b15640 1 -- 172.21.15.104:0/2697329516 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe0e4005330 con 0x7fe1040b04e0 2024-03-29T11:20:41.412 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.435+0000 7fe0ebfff640 1 -- 172.21.15.104:0/2697329516 <== mon.1 v1:172.21.15.119:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7fe0fc0530b0 con 0x7fe1040b04e0 2024-03-29T11:20:41.412 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.435+0000 7fe0ebfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:41.412 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.435+0000 7fe0ebfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:41.412 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:20:41.412 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:41.415 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.439+0000 7fe109b15640 10 librados: watch_flush enter 2024-03-29T11:20:41.415 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.439+0000 7fe109b15640 10 librados: watch_flush exit 2024-03-29T11:20:41.415 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.439+0000 7fe109b15640 20 client.4267.objecter shutdown clearing up homeless session... 2024-03-29T11:20:41.416 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.439+0000 7fe109b15640 10 client.4267.objecter successfully canceled tick 2024-03-29T11:20:41.416 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.439+0000 7fe109b15640 1 -- 172.21.15.104:0/2697329516 >> v1:172.21.15.104:6816/3341126660 conn(0x7fe0f006dbb0 legacy=0x7fe0f0070070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:41.416 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.439+0000 7fe109b15640 10 monclient: shutdown 2024-03-29T11:20:41.416 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.439+0000 7fe109b15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:41.417 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.439+0000 7fe109b15640 1 -- 172.21.15.104:0/2697329516 >> v1:172.21.15.119:6789/0 conn(0x7fe1040b04e0 legacy=0x7fe1041a2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:41.417 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.439+0000 7fe109b15640 1 -- 172.21.15.104:0/2697329516 shutdown_connections 2024-03-29T11:20:41.417 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.439+0000 7fe109b15640 1 -- 172.21.15.104:0/2697329516 >> 172.21.15.104:0/2697329516 conn(0x7fe1040672a0 msgr2=0x7fe10406b400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:41.417 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.440+0000 7fe109b15640 1 -- 172.21.15.104:0/2697329516 shutdown_connections 2024-03-29T11:20:41.417 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.440+0000 7fe109b15640 1 -- 172.21.15.104:0/2697329516 wait complete. 2024-03-29T11:20:41.417 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:41.440+0000 7fe109b15640 1 librados: shutdown 2024-03-29T11:20:41.432 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:48.240 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:48.373 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.397+0000 7f7bb5f25640 10 monclient: get_monmap_and_config 2024-03-29T11:20:48.373 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.397+0000 7f7bb5f25640 10 monclient: build_initial_monmap 2024-03-29T11:20:48.374 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.397+0000 7f7bb5f25640 10 monclient: monmap: 2024-03-29T11:20:48.374 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:20:48.374 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:20:48.374 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:20:48.397948+0000 2024-03-29T11:20:48.374 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:20:48.397948+0000 2024-03-29T11:20:48.374 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:48.374 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:20:48.374 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:20:48.374 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:20:48.374 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:20:48.374 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:20:48.375 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.397+0000 7f7bb5f25640 1 Processor -- start 2024-03-29T11:20:48.375 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.398+0000 7f7bb5f25640 1 -- start start 2024-03-29T11:20:48.375 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.398+0000 7f7bb5f25640 10 monclient: init 2024-03-29T11:20:48.375 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.398+0000 7f7bb5f25640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:48.376 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.398+0000 7f7bb5f25640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:20:48.376 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.399+0000 7f7bb5f25640 10 monclient(hunting): picked mon.noname-a con 0x7f7bb00b0770 addr v1:172.21.15.104:6789/0 2024-03-29T11:20:48.376 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.399+0000 7f7bb5f25640 10 monclient(hunting): picked mon.noname-b con 0x7f7bb014a700 addr v1:172.21.15.104:6790/0 2024-03-29T11:20:48.376 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.399+0000 7f7bb5f25640 10 monclient(hunting): picked mon.noname-c con 0x7f7bb00ac5a0 addr v1:172.21.15.119:6789/0 2024-03-29T11:20:48.376 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.399+0000 7f7bb5f25640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bb00aba70 con 0x7f7bb00b0770 2024-03-29T11:20:48.376 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.399+0000 7f7bb5f25640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bb00aecb0 con 0x7f7bb00ac5a0 2024-03-29T11:20:48.376 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.399+0000 7f7bb5f25640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bb00af150 con 0x7f7bb014a700 2024-03-29T11:20:48.376 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.399+0000 7f7bb5f25640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:48.376 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.399+0000 7f7bb5f25640 10 monclient(hunting): authenticate will time out at 1029.650160s 2024-03-29T11:20:48.376 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.399+0000 7f7baffff640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f7bb00ac5a0 0x7f7bb0151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:59446/0 (socket says 172.21.15.104:59446) 2024-03-29T11:20:48.376 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.399+0000 7f7baffff640 1 -- 172.21.15.104:0/1496264476 learned_addr learned my addr 172.21.15.104:0/1496264476 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:20:48.377 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.399+0000 7f7bae7fc640 1 -- 172.21.15.104:0/1496264476 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3233434335 0 0) 0x7f7bb00aecb0 con 0x7f7bb00ac5a0 2024-03-29T11:20:48.377 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.399+0000 7f7bae7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:48.377 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.399+0000 7f7bae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:48.377 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.399+0000 7f7bae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:48.377 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.399+0000 7f7bae7fc640 10 monclient(hunting): my global_id is 4270 2024-03-29T11:20:48.377 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.399+0000 7f7bae7fc640 1 -- 172.21.15.104:0/1496264476 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b98003520 con 0x7f7bb00ac5a0 2024-03-29T11:20:48.377 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.400+0000 7f7bae7fc640 1 -- 172.21.15.104:0/1496264476 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4161426144 0 0) 0x7f7b98003520 con 0x7f7bb00ac5a0 2024-03-29T11:20:48.377 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.400+0000 7f7bae7fc640 1 -- 172.21.15.104:0/1496264476 >> v1:172.21.15.104:6790/0 conn(0x7f7bb014a700 legacy=0x7f7bb00abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:48.377 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.400+0000 7f7bae7fc640 1 -- 172.21.15.104:0/1496264476 >> v1:172.21.15.104:6789/0 conn(0x7f7bb00b0770 legacy=0x7f7bb0149ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:20:48.378 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.401+0000 7f7bae7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:20:48.378 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.401+0000 7f7bae7fc640 1 monclient: found mon.noname-c 2024-03-29T11:20:48.378 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.401+0000 7f7bae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:48.379 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.401+0000 7f7bae7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.119:6789/0 2024-03-29T11:20:48.379 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.401+0000 7f7bae7fc640 1 -- 172.21.15.104:0/1496264476 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bb00ac9d0 con 0x7f7bb00ac5a0 2024-03-29T11:20:48.379 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.401+0000 7f7bae7fc640 10 monclient: _finish_auth 0 2024-03-29T11:20:48.379 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.401+0000 7f7bae7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:20:48.379 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.401+0000 7f7bae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:48.379 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.401+0000 7f7bb5f25640 5 monclient: authenticate success, global_id 4270 2024-03-29T11:20:48.379 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.401+0000 7f7bae7fc640 1 -- 172.21.15.104:0/1496264476 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f7ba8001a30 con 0x7f7bb00ac5a0 2024-03-29T11:20:48.379 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.401+0000 7f7bae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:48.379 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.401+0000 7f7bae7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:48.379 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.401+0000 7f7bae7fc640 10 monclient: dump: 2024-03-29T11:20:48.380 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:20:48.380 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:20:48.380 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:48.380 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:48.380 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:48.380 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:20:48.380 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:20:48.380 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:20:48.380 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:20:48.380 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:20:48.380 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.401+0000 7f7bb5f25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:48.380 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.402+0000 7f7bae7fc640 1 -- 172.21.15.104:0/1496264476 <== mon.1 v1:172.21.15.119:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ba8001c00 con 0x7f7bb00ac5a0 2024-03-29T11:20:48.381 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.402+0000 7f7bae7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:48.381 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.402+0000 7f7bae7fc640 1 -- 172.21.15.104:0/1496264476 <== mon.1 v1:172.21.15.119:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f7ba8002010 con 0x7f7bb00ac5a0 2024-03-29T11:20:48.381 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.402+0000 7f7bb5f25640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:48.381 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.402+0000 7f7bae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:48.381 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.402+0000 7f7bae7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:48.381 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.402+0000 7f7bae7fc640 10 monclient: dump: 2024-03-29T11:20:48.381 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:20:48.381 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:20:48.381 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:48.381 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:48.382 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:48.382 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:20:48.382 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:20:48.382 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:20:48.382 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:20:48.382 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:20:48.382 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.402+0000 7f7bb5f25640 4 set_mon_vals no callback set 2024-03-29T11:20:48.391 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.403+0000 7f7bb5f25640 10 monclient: shutdown 2024-03-29T11:20:48.391 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.403+0000 7f7bb5f25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:48.391 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.403+0000 7f7bb5f25640 1 -- 172.21.15.104:0/1496264476 >> v1:172.21.15.119:6789/0 conn(0x7f7bb00ac5a0 legacy=0x7f7bb0151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:48.391 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.403+0000 7f7bb5f25640 1 -- 172.21.15.104:0/1496264476 shutdown_connections 2024-03-29T11:20:48.391 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.403+0000 7f7bb5f25640 1 -- 172.21.15.104:0/1496264476 >> 172.21.15.104:0/1496264476 conn(0x7f7bb006b660 msgr2=0x7f7bb006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:48.391 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.403+0000 7f7bb5f25640 1 -- 172.21.15.104:0/1496264476 shutdown_connections 2024-03-29T11:20:48.391 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.403+0000 7f7bb5f25640 1 -- 172.21.15.104:0/1496264476 wait complete. 2024-03-29T11:20:48.392 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.404+0000 7f7bb5f25640 10 monclient: build_initial_monmap 2024-03-29T11:20:48.392 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.404+0000 7f7bb5f25640 10 monclient: monmap: 2024-03-29T11:20:48.392 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:20:48.392 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:48.392 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:20:48.392 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:20:48.392 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:48.392 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:20:48.392 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:20:48.392 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:20:48.392 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:20:48.392 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:20:48.392 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.404+0000 7f7bb5f25640 1 librados: starting msgr at 2024-03-29T11:20:48.392 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.404+0000 7f7bb5f25640 1 librados: starting objecter 2024-03-29T11:20:48.393 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.404+0000 7f7bb5f25640 1 Processor -- start 2024-03-29T11:20:48.393 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.405+0000 7f7bb5f25640 1 -- start start 2024-03-29T11:20:48.393 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.405+0000 7f7bb5f25640 1 librados: setting wanted keys 2024-03-29T11:20:48.393 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.405+0000 7f7bb5f25640 1 librados: calling monclient init 2024-03-29T11:20:48.393 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.405+0000 7f7bb5f25640 10 monclient: init 2024-03-29T11:20:48.393 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.406+0000 7f7bb5f25640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:48.394 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.406+0000 7f7bb5f25640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:20:48.394 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.406+0000 7f7bb5f25640 10 monclient(hunting): picked mon.noname-c con 0x7f7bb00ac5a0 addr v1:172.21.15.104:6790/0 2024-03-29T11:20:48.394 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.406+0000 7f7baf7fe640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f7bb00ac5a0 0x7f7bb01db9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:56734/0 (socket says 172.21.15.104:56734) 2024-03-29T11:20:48.394 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.406+0000 7f7baf7fe640 1 -- 172.21.15.104:0/2073936452 learned_addr learned my addr 172.21.15.104:0/2073936452 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:20:48.394 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.406+0000 7f7b97fff640 10 client.?.objecter ms_handle_connect 0x7f7bb00ac5a0 2024-03-29T11:20:48.394 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.406+0000 7f7b97fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:48.394 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.406+0000 7f7bb5f25640 10 monclient(hunting): picked mon.noname-a con 0x7f7bb00b0770 addr v1:172.21.15.104:6789/0 2024-03-29T11:20:48.394 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.407+0000 7f7bb5f25640 10 monclient(hunting): picked mon.noname-b con 0x7f7bb014a700 addr v1:172.21.15.119:6789/0 2024-03-29T11:20:48.394 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.407+0000 7f7bb5f25640 1 -- 172.21.15.104:0/2073936452 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bb0125690 con 0x7f7bb00b0770 2024-03-29T11:20:48.394 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.407+0000 7f7bb5f25640 1 -- 172.21.15.104:0/2073936452 --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bb012a0c0 con 0x7f7bb014a700 2024-03-29T11:20:48.394 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.407+0000 7f7bb5f25640 1 -- 172.21.15.104:0/2073936452 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bb0147d30 con 0x7f7bb00ac5a0 2024-03-29T11:20:48.394 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.407+0000 7f7b97fff640 10 client.?.objecter ms_handle_connect 0x7f7bb014a700 2024-03-29T11:20:48.394 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.407+0000 7f7b97fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:48.395 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.407+0000 7f7bb5f25640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:48.395 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.407+0000 7f7b97fff640 1 -- 172.21.15.104:0/2073936452 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1099525764 0 0) 0x7f7bb012a0c0 con 0x7f7bb014a700 2024-03-29T11:20:48.395 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.407+0000 7f7bb5f25640 10 monclient(hunting): authenticate will time out at 1029.658733s 2024-03-29T11:20:48.395 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.407+0000 7f7b97fff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:48.395 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.407+0000 7f7b97fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:48.395 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.407+0000 7f7b97fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:48.395 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.407+0000 7f7b97fff640 10 monclient(hunting): my global_id is 4273 2024-03-29T11:20:48.395 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.407+0000 7f7b97fff640 1 -- 172.21.15.104:0/2073936452 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b9c004eb0 con 0x7f7bb014a700 2024-03-29T11:20:48.395 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.407+0000 7f7b97fff640 1 -- 172.21.15.104:0/2073936452 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3307220495 0 0) 0x7f7bb0147d30 con 0x7f7bb00ac5a0 2024-03-29T11:20:48.395 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.407+0000 7f7b97fff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:48.395 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.407+0000 7f7b97fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:48.395 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.407+0000 7f7b97fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:48.395 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.407+0000 7f7b97fff640 10 monclient(hunting): my global_id is 4295 2024-03-29T11:20:48.395 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.407+0000 7f7b97fff640 1 -- 172.21.15.104:0/2073936452 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bb012a0c0 con 0x7f7bb00ac5a0 2024-03-29T11:20:48.396 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.408+0000 7f7b97fff640 1 -- 172.21.15.104:0/2073936452 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2086695727 0 0) 0x7f7bb012a0c0 con 0x7f7bb00ac5a0 2024-03-29T11:20:48.396 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.408+0000 7f7b97fff640 1 -- 172.21.15.104:0/2073936452 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7bb0147d30 con 0x7f7bb00ac5a0 2024-03-29T11:20:48.396 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.408+0000 7f7b97fff640 1 -- 172.21.15.104:0/2073936452 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f7ba0001920 con 0x7f7bb00ac5a0 2024-03-29T11:20:48.396 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.408+0000 7f7b97fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:48.396 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.408+0000 7f7b97fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:48.396 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.408+0000 7f7b97fff640 10 monclient(hunting): dump: 2024-03-29T11:20:48.396 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:20:48.396 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:20:48.396 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:48.396 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:48.396 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:48.397 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:20:48.397 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:20:48.397 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:20:48.397 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:20:48.397 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:20:48.397 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.408+0000 7f7b97fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:48.397 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.408+0000 7f7b97fff640 1 -- 172.21.15.104:0/2073936452 <== mon.2 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3196112994 0 0) 0x7f7bb0147d30 con 0x7f7bb00ac5a0 2024-03-29T11:20:48.397 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.408+0000 7f7b97fff640 1 -- 172.21.15.104:0/2073936452 >> v1:172.21.15.119:6789/0 conn(0x7f7bb014a700 legacy=0x7f7bb0147620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:48.397 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.409+0000 7f7b97fff640 1 -- 172.21.15.104:0/2073936452 >> v1:172.21.15.104:6789/0 conn(0x7f7bb00b0770 legacy=0x7f7bb0146f10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:20:48.397 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.409+0000 7f7b97fff640 10 monclient: _finish_hunting 0 2024-03-29T11:20:48.397 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.409+0000 7f7b97fff640 1 monclient: found mon.c 2024-03-29T11:20:48.397 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.409+0000 7f7b97fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:48.398 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.409+0000 7f7b97fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:20:48.398 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.409+0000 7f7b97fff640 1 -- 172.21.15.104:0/2073936452 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bb012a330 con 0x7f7bb00ac5a0 2024-03-29T11:20:48.398 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.409+0000 7f7b97fff640 10 monclient: _finish_auth 0 2024-03-29T11:20:48.398 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.409+0000 7f7b97fff640 10 monclient: _check_auth_tickets 2024-03-29T11:20:48.398 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.409+0000 7f7b97fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:48.398 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.409+0000 7f7bb5f25640 5 monclient: authenticate success, global_id 4295 2024-03-29T11:20:48.398 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.409+0000 7f7bb5f25640 10 monclient: _renew_subs 2024-03-29T11:20:48.398 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.409+0000 7f7bb5f25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:20:48.398 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.409+0000 7f7bb5f25640 1 -- 172.21.15.104:0/2073936452 --> v1:172.21.15.104:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bb012a610 con 0x7f7bb00ac5a0 2024-03-29T11:20:48.398 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.409+0000 7f7b97fff640 1 -- 172.21.15.104:0/2073936452 <== mon.2 v1:172.21.15.104:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ba0001a80 con 0x7f7bb00ac5a0 2024-03-29T11:20:48.398 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.409+0000 7f7b97fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:48.398 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.409+0000 7f7b97fff640 1 -- 172.21.15.104:0/2073936452 <== mon.2 v1:172.21.15.104:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f7ba00038a0 con 0x7f7bb00ac5a0 2024-03-29T11:20:48.398 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.409+0000 7f7b97fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:48.399 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.409+0000 7f7b97fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:48.399 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.409+0000 7f7b97fff640 10 monclient: dump: 2024-03-29T11:20:48.399 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:20:48.399 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:20:48.399 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:48.399 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:48.399 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:48.399 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:20:48.399 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:20:48.399 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:20:48.399 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:20:48.399 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:20:48.399 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.409+0000 7f7bb5f25640 10 client.4295.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:48.399 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.409+0000 7f7bb5f25640 10 monclient: _renew_subs 2024-03-29T11:20:48.400 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.409+0000 7f7bb5f25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:20:48.400 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.409+0000 7f7bb5f25640 1 -- 172.21.15.104:0/2073936452 --> v1:172.21.15.104:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bb0128400 con 0x7f7bb00ac5a0 2024-03-29T11:20:48.400 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.409+0000 7f7badffb640 4 set_mon_vals no callback set 2024-03-29T11:20:48.400 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.409+0000 7f7bb5f25640 1 librados: init done 2024-03-29T11:20:48.400 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.411+0000 7f7bb5f25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:48.400 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.411+0000 7f7bb5f25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:48.400 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.411+0000 7f7bb5f25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:20:48.400 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.411+0000 7f7bb5f25640 1 -- 172.21.15.104:0/2073936452 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bb0075f10 con 0x7f7bb00ac5a0 2024-03-29T11:20:48.401 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.411+0000 7f7b97fff640 1 -- 172.21.15.104:0/2073936452 <== mon.2 v1:172.21.15.104:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f7ba0003a00 con 0x7f7bb00ac5a0 2024-03-29T11:20:48.401 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.412+0000 7f7b97fff640 1 -- 172.21.15.104:0/2073936452 <== mon.2 v1:172.21.15.104:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f7ba00849c0 con 0x7f7bb00ac5a0 2024-03-29T11:20:48.401 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.412+0000 7f7b97fff640 10 client.4295.objecter ms_dispatch 0x7f7bb0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:48.401 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.412+0000 7f7b97fff640 3 client.4295.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:48.401 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.412+0000 7f7b97fff640 3 client.4295.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:48.401 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.412+0000 7f7b97fff640 20 client.4295.objecter dump_active .. 0 homeless 2024-03-29T11:20:48.401 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.412+0000 7f7baeffd640 10 monclient: get_auth_request con 0x7f7b9c06ddd0 auth_method 0 2024-03-29T11:20:48.401 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.412+0000 7f7b97fff640 10 client.4295.objecter ms_handle_connect 0x7f7b9c06ddd0 2024-03-29T11:20:48.402 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.419+0000 7f7b97fff640 1 -- 172.21.15.104:0/2073936452 <== mon.2 v1:172.21.15.104:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7ba00500d0 con 0x7f7bb00ac5a0 2024-03-29T11:20:48.402 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.419+0000 7f7b97fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:48.402 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.419+0000 7f7b97fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:48.704 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.728+0000 7f7bb5f25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:48.704 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.728+0000 7f7bb5f25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:48.704 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.728+0000 7f7bb5f25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:20:48.704 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.728+0000 7f7bb5f25640 1 -- 172.21.15.104:0/2073936452 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7bb006ab70 con 0x7f7bb00ac5a0 2024-03-29T11:20:48.705 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.729+0000 7f7b97fff640 1 -- 172.21.15.104:0/2073936452 <== mon.2 v1:172.21.15.104:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f7ba00521d0 con 0x7f7bb00ac5a0 2024-03-29T11:20:48.705 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.729+0000 7f7b97fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:48.705 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.729+0000 7f7b97fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:48.708 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:20:48.708 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:48.709 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.733+0000 7f7bb5f25640 10 librados: watch_flush enter 2024-03-29T11:20:48.710 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.733+0000 7f7bb5f25640 10 librados: watch_flush exit 2024-03-29T11:20:48.710 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.733+0000 7f7bb5f25640 20 client.4295.objecter shutdown clearing up homeless session... 2024-03-29T11:20:48.710 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.733+0000 7f7bb5f25640 10 client.4295.objecter successfully canceled tick 2024-03-29T11:20:48.710 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.733+0000 7f7bb5f25640 1 -- 172.21.15.104:0/2073936452 >> v1:172.21.15.104:6816/3341126660 conn(0x7f7b9c06ddd0 legacy=0x7f7b9c070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:48.710 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.733+0000 7f7bb5f25640 10 monclient: shutdown 2024-03-29T11:20:48.710 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.733+0000 7f7bb5f25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:48.710 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.733+0000 7f7bb5f25640 1 -- 172.21.15.104:0/2073936452 >> v1:172.21.15.104:6790/0 conn(0x7f7bb00ac5a0 legacy=0x7f7bb01db9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:48.710 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.733+0000 7f7bb5f25640 1 -- 172.21.15.104:0/2073936452 shutdown_connections 2024-03-29T11:20:48.710 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.733+0000 7f7bb5f25640 1 -- 172.21.15.104:0/2073936452 >> 172.21.15.104:0/2073936452 conn(0x7f7bb006b660 msgr2=0x7f7bb01494a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:48.711 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.734+0000 7f7bb5f25640 1 -- 172.21.15.104:0/2073936452 shutdown_connections 2024-03-29T11:20:48.711 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.734+0000 7f7bb5f25640 1 -- 172.21.15.104:0/2073936452 wait complete. 2024-03-29T11:20:48.711 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:48.734+0000 7f7bb5f25640 1 librados: shutdown 2024-03-29T11:20:48.724 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:55.532 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:55.679 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.702+0000 7fb31a4b3640 10 monclient: get_monmap_and_config 2024-03-29T11:20:55.679 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.702+0000 7fb31a4b3640 10 monclient: build_initial_monmap 2024-03-29T11:20:55.679 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.702+0000 7fb31a4b3640 10 monclient: monmap: 2024-03-29T11:20:55.679 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:20:55.679 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:20:55.680 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:20:55.703207+0000 2024-03-29T11:20:55.680 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:20:55.703207+0000 2024-03-29T11:20:55.680 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:55.680 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:20:55.680 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:20:55.680 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:20:55.680 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:20:55.681 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:20:55.681 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.703+0000 7fb31a4b3640 1 Processor -- start 2024-03-29T11:20:55.682 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.703+0000 7fb31a4b3640 1 -- start start 2024-03-29T11:20:55.682 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.703+0000 7fb31a4b3640 10 monclient: init 2024-03-29T11:20:55.682 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.704+0000 7fb31a4b3640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:55.682 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.704+0000 7fb31a4b3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:20:55.683 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.705+0000 7fb31a4b3640 10 monclient(hunting): picked mon.noname-b con 0x7fb3140acf60 addr v1:172.21.15.104:6790/0 2024-03-29T11:20:55.683 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.705+0000 7fb31a4b3640 10 monclient(hunting): picked mon.noname-a con 0x7fb31413ff40 addr v1:172.21.15.104:6789/0 2024-03-29T11:20:55.683 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.705+0000 7fb31a4b3640 10 monclient(hunting): picked mon.noname-c con 0x7fb3141403f0 addr v1:172.21.15.119:6789/0 2024-03-29T11:20:55.683 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.705+0000 7fb31a4b3640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3140b4310 con 0x7fb31413ff40 2024-03-29T11:20:55.683 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.705+0000 7fb31a4b3640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb314140800 con 0x7fb3141403f0 2024-03-29T11:20:55.683 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.705+0000 7fb31a4b3640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3140aec00 con 0x7fb3140acf60 2024-03-29T11:20:55.683 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.705+0000 7fb31a4b3640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:55.683 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.705+0000 7fb31a4b3640 10 monclient(hunting): authenticate will time out at 1036.956112s 2024-03-29T11:20:55.684 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.707+0000 7fb313fff640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7fb3140acf60 0x7fb314139b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:58617/0 (socket says 172.21.15.104:58617) 2024-03-29T11:20:55.684 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.707+0000 7fb313fff640 1 -- 172.21.15.104:0/260438959 learned_addr learned my addr 172.21.15.104:0/260438959 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:20:55.685 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.708+0000 7fb3137fe640 1 -- 172.21.15.104:0/260438959 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2770779357 0 0) 0x7fb3140aec00 con 0x7fb3140acf60 2024-03-29T11:20:55.685 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.708+0000 7fb3137fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:55.685 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.708+0000 7fb3137fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:55.685 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.708+0000 7fb3137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:55.685 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.708+0000 7fb3137fe640 10 monclient(hunting): my global_id is 4298 2024-03-29T11:20:55.686 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.708+0000 7fb3137fe640 1 -- 172.21.15.104:0/260438959 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb300003520 con 0x7fb3140acf60 2024-03-29T11:20:55.686 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.708+0000 7fb3137fe640 1 -- 172.21.15.104:0/260438959 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 278447599 0 0) 0x7fb314140800 con 0x7fb3141403f0 2024-03-29T11:20:55.686 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.708+0000 7fb3137fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:55.686 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.708+0000 7fb3137fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:55.686 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.708+0000 7fb3137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:55.686 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.708+0000 7fb3137fe640 10 monclient(hunting): my global_id is 4276 2024-03-29T11:20:55.686 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.708+0000 7fb3137fe640 1 -- 172.21.15.104:0/260438959 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3140aec00 con 0x7fb3141403f0 2024-03-29T11:20:55.686 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.708+0000 7fb3137fe640 1 -- 172.21.15.104:0/260438959 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1320678810 0 0) 0x7fb300003520 con 0x7fb3140acf60 2024-03-29T11:20:55.687 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.708+0000 7fb3137fe640 1 -- 172.21.15.104:0/260438959 >> v1:172.21.15.119:6789/0 conn(0x7fb3141403f0 legacy=0x7fb31414bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:55.687 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.708+0000 7fb3137fe640 1 -- 172.21.15.104:0/260438959 >> v1:172.21.15.104:6789/0 conn(0x7fb31413ff40 legacy=0x7fb314148840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:20:55.689 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.710+0000 7fb3137fe640 10 monclient: _finish_hunting 0 2024-03-29T11:20:55.690 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.710+0000 7fb3137fe640 1 monclient: found mon.noname-b 2024-03-29T11:20:55.690 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.710+0000 7fb3137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:55.690 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.710+0000 7fb3137fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.104:6790/0 2024-03-29T11:20:55.690 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.710+0000 7fb3137fe640 1 -- 172.21.15.104:0/260438959 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3140aeee0 con 0x7fb3140acf60 2024-03-29T11:20:55.690 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.710+0000 7fb3137fe640 10 monclient: _finish_auth 0 2024-03-29T11:20:55.690 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.710+0000 7fb3137fe640 10 monclient: _check_auth_tickets 2024-03-29T11:20:55.690 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.710+0000 7fb3137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:55.690 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.710+0000 7fb3137fe640 1 -- 172.21.15.104:0/260438959 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fb30c001a10 con 0x7fb3140acf60 2024-03-29T11:20:55.690 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.710+0000 7fb3137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:55.691 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.710+0000 7fb3137fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:55.691 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.710+0000 7fb3137fe640 10 monclient: dump: 2024-03-29T11:20:55.691 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:20:55.691 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:20:55.691 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:55.691 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:55.691 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:55.691 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:20:55.691 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:20:55.692 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:20:55.692 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:20:55.692 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:20:55.692 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.710+0000 7fb31a4b3640 5 monclient: authenticate success, global_id 4298 2024-03-29T11:20:55.692 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.710+0000 7fb31a4b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:55.692 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.710+0000 7fb3137fe640 1 -- 172.21.15.104:0/260438959 <== mon.2 v1:172.21.15.104:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb30c001c30 con 0x7fb3140acf60 2024-03-29T11:20:55.692 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.710+0000 7fb3137fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:55.692 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.710+0000 7fb3137fe640 1 -- 172.21.15.104:0/260438959 <== mon.2 v1:172.21.15.104:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fb30c001fc0 con 0x7fb3140acf60 2024-03-29T11:20:55.692 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.710+0000 7fb3137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:55.692 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.710+0000 7fb3137fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:55.693 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.710+0000 7fb3137fe640 10 monclient: dump: 2024-03-29T11:20:55.693 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:20:55.693 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:20:55.693 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:55.693 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:55.693 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:55.693 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:20:55.693 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:20:55.693 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:20:55.693 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:20:55.693 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:20:55.693 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.710+0000 7fb31a4b3640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:55.693 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.710+0000 7fb31a4b3640 4 set_mon_vals no callback set 2024-03-29T11:20:55.693 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.711+0000 7fb31a4b3640 10 monclient: shutdown 2024-03-29T11:20:55.693 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.711+0000 7fb31a4b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:55.694 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.711+0000 7fb31a4b3640 1 -- 172.21.15.104:0/260438959 >> v1:172.21.15.104:6790/0 conn(0x7fb3140acf60 legacy=0x7fb314139b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:55.694 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.711+0000 7fb31a4b3640 1 -- 172.21.15.104:0/260438959 shutdown_connections 2024-03-29T11:20:55.694 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.711+0000 7fb31a4b3640 1 -- 172.21.15.104:0/260438959 >> 172.21.15.104:0/260438959 conn(0x7fb3140b35e0 msgr2=0x7fb3140b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:55.694 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.711+0000 7fb31a4b3640 1 -- 172.21.15.104:0/260438959 shutdown_connections 2024-03-29T11:20:55.694 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.711+0000 7fb31a4b3640 1 -- 172.21.15.104:0/260438959 wait complete. 2024-03-29T11:20:55.694 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.712+0000 7fb31a4b3640 10 monclient: build_initial_monmap 2024-03-29T11:20:55.694 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.712+0000 7fb31a4b3640 10 monclient: monmap: 2024-03-29T11:20:55.694 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:20:55.694 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:55.694 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:20:55.694 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:20:55.694 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:55.694 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:20:55.694 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:20:55.695 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:20:55.695 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:20:55.695 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:20:55.695 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.712+0000 7fb31a4b3640 1 librados: starting msgr at 2024-03-29T11:20:55.695 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.712+0000 7fb31a4b3640 1 librados: starting objecter 2024-03-29T11:20:55.695 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.713+0000 7fb31a4b3640 1 Processor -- start 2024-03-29T11:20:55.695 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.713+0000 7fb31a4b3640 1 -- start start 2024-03-29T11:20:55.695 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.713+0000 7fb31a4b3640 1 librados: setting wanted keys 2024-03-29T11:20:55.695 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.713+0000 7fb31a4b3640 1 librados: calling monclient init 2024-03-29T11:20:55.695 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.713+0000 7fb31a4b3640 10 monclient: init 2024-03-29T11:20:55.698 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.714+0000 7fb31a4b3640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:55.698 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.714+0000 7fb31a4b3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:20:55.699 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.714+0000 7fb31a4b3640 10 monclient(hunting): picked mon.noname-a con 0x7fb3140acf60 addr v1:172.21.15.104:6789/0 2024-03-29T11:20:55.699 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.715+0000 7fb31a4b3640 10 monclient(hunting): picked mon.noname-c con 0x7fb31413ff40 addr v1:172.21.15.104:6790/0 2024-03-29T11:20:55.699 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.715+0000 7fb31a4b3640 10 monclient(hunting): picked mon.noname-b con 0x7fb3141403f0 addr v1:172.21.15.119:6789/0 2024-03-29T11:20:55.699 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.715+0000 7fb31a4b3640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb31413e950 con 0x7fb3140acf60 2024-03-29T11:20:55.699 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.715+0000 7fb31a4b3640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb31413eba0 con 0x7fb3141403f0 2024-03-29T11:20:55.699 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.715+0000 7fb31a4b3640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb31413f7d0 con 0x7fb31413ff40 2024-03-29T11:20:55.699 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.715+0000 7fb31a4b3640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:55.699 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.715+0000 7fb31a4b3640 10 monclient(hunting): authenticate will time out at 1036.966375s 2024-03-29T11:20:55.699 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.715+0000 7fb318a50640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fb3141403f0 0x7fb3141d5750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:60946/0 (socket says 172.21.15.104:60946) 2024-03-29T11:20:55.699 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.715+0000 7fb318a50640 1 -- 172.21.15.104:0/2133959057 learned_addr learned my addr 172.21.15.104:0/2133959057 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:20:55.699 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.715+0000 7fb3117fa640 10 client.?.objecter ms_handle_connect 0x7fb3141403f0 2024-03-29T11:20:55.699 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.716+0000 7fb3117fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:55.699 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.716+0000 7fb3117fa640 1 -- 172.21.15.104:0/2133959057 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 386475654 0 0) 0x7fb31413eba0 con 0x7fb3141403f0 2024-03-29T11:20:55.699 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.716+0000 7fb3117fa640 10 monclient(hunting): _negotiate 2024-03-29T11:20:55.700 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.716+0000 7fb3117fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:55.700 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.716+0000 7fb3117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:55.700 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.716+0000 7fb3117fa640 10 monclient(hunting): my global_id is 4279 2024-03-29T11:20:55.700 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.716+0000 7fb3117fa640 1 -- 172.21.15.104:0/2133959057 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2f4004eb0 con 0x7fb3141403f0 2024-03-29T11:20:55.700 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.716+0000 7fb3117fa640 1 -- 172.21.15.104:0/2133959057 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3524637928 0 0) 0x7fb2f4004eb0 con 0x7fb3141403f0 2024-03-29T11:20:55.700 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.716+0000 7fb3117fa640 1 -- 172.21.15.104:0/2133959057 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb31413eba0 con 0x7fb3141403f0 2024-03-29T11:20:55.700 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.716+0000 7fb3117fa640 1 -- 172.21.15.104:0/2133959057 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fb304001830 con 0x7fb3141403f0 2024-03-29T11:20:55.700 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.716+0000 7fb3117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:55.700 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.716+0000 7fb3117fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:55.700 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.716+0000 7fb3117fa640 10 monclient(hunting): dump: 2024-03-29T11:20:55.700 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:20:55.700 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:20:55.700 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:55.700 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:55.700 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:55.701 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:20:55.701 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:20:55.701 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:20:55.701 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:20:55.701 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:20:55.701 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.716+0000 7fb3117fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:55.701 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.717+0000 7fb3117fa640 1 -- 172.21.15.104:0/2133959057 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 705851605 0 0) 0x7fb31413eba0 con 0x7fb3141403f0 2024-03-29T11:20:55.701 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.717+0000 7fb3117fa640 1 -- 172.21.15.104:0/2133959057 >> v1:172.21.15.104:6790/0 conn(0x7fb31413ff40 legacy=0x7fb3141d5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:55.701 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.717+0000 7fb3117fa640 1 -- 172.21.15.104:0/2133959057 >> v1:172.21.15.104:6789/0 conn(0x7fb3140acf60 legacy=0x7fb31413f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:55.701 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.717+0000 7fb3117fa640 10 monclient: _finish_hunting 0 2024-03-29T11:20:55.701 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.717+0000 7fb3117fa640 1 monclient: found mon.b 2024-03-29T11:20:55.701 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.717+0000 7fb3117fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:55.701 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.717+0000 7fb3117fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:20:55.701 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.717+0000 7fb3117fa640 1 -- 172.21.15.104:0/2133959057 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb31413f960 con 0x7fb3141403f0 2024-03-29T11:20:55.701 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.717+0000 7fb3117fa640 10 monclient: _finish_auth 0 2024-03-29T11:20:55.702 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.717+0000 7fb3117fa640 10 monclient: _check_auth_tickets 2024-03-29T11:20:55.702 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.717+0000 7fb3117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:55.702 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.717+0000 7fb3117fa640 10 client.?.objecter ms_handle_connect 0x7fb3140acf60 2024-03-29T11:20:55.702 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.717+0000 7fb3117fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:55.702 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.717+0000 7fb31a4b3640 5 monclient: authenticate success, global_id 4279 2024-03-29T11:20:55.702 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.717+0000 7fb31a4b3640 10 monclient: _renew_subs 2024-03-29T11:20:55.702 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.717+0000 7fb31a4b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:20:55.702 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.717+0000 7fb31a4b3640 1 -- 172.21.15.104:0/2133959057 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3141375b0 con 0x7fb3141403f0 2024-03-29T11:20:55.702 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.717+0000 7fb31a4b3640 10 client.4279.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:55.703 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.717+0000 7fb31a4b3640 10 monclient: _renew_subs 2024-03-29T11:20:55.703 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.717+0000 7fb31a4b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:20:55.703 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.717+0000 7fb31a4b3640 1 -- 172.21.15.104:0/2133959057 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3141378b0 con 0x7fb3141403f0 2024-03-29T11:20:55.703 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.717+0000 7fb31a4b3640 1 librados: init done 2024-03-29T11:20:55.703 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.717+0000 7fb3117fa640 1 -- 172.21.15.104:0/2133959057 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb304001990 con 0x7fb3141403f0 2024-03-29T11:20:55.703 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.717+0000 7fb3117fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:55.703 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.717+0000 7fb3117fa640 1 -- 172.21.15.104:0/2133959057 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fb304003870 con 0x7fb3141403f0 2024-03-29T11:20:55.703 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.717+0000 7fb3117fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:55.703 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.717+0000 7fb3117fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:55.703 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.717+0000 7fb3117fa640 10 monclient: dump: 2024-03-29T11:20:55.703 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:20:55.703 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:20:55.704 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:55.704 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:20:55.704 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:55.704 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:20:55.704 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:20:55.704 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:20:55.704 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:20:55.704 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:20:55.704 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.717+0000 7fb312ffd640 4 set_mon_vals no callback set 2024-03-29T11:20:55.704 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.718+0000 7fb31a4b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:55.704 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.718+0000 7fb31a4b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:55.704 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.718+0000 7fb31a4b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:20:55.705 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.718+0000 7fb31a4b3640 1 -- 172.21.15.104:0/2133959057 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb314075f10 con 0x7fb3141403f0 2024-03-29T11:20:55.705 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.719+0000 7fb3117fa640 1 -- 172.21.15.104:0/2133959057 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7fb3040039d0 con 0x7fb3141403f0 2024-03-29T11:20:55.705 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.720+0000 7fb313fff640 10 monclient: get_auth_request con 0x7fb2f406dc90 auth_method 0 2024-03-29T11:20:55.705 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.721+0000 7fb3117fa640 10 client.4279.objecter ms_handle_connect 0x7fb2f406dc90 2024-03-29T11:20:55.705 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.721+0000 7fb3117fa640 1 -- 172.21.15.104:0/2133959057 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7fb304084980 con 0x7fb3141403f0 2024-03-29T11:20:55.705 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.721+0000 7fb3117fa640 10 client.4279.objecter ms_dispatch 0x7fb314000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:55.705 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.721+0000 7fb3117fa640 3 client.4279.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:55.705 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.721+0000 7fb3117fa640 3 client.4279.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:55.705 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.721+0000 7fb3117fa640 20 client.4279.objecter dump_active .. 0 homeless 2024-03-29T11:20:55.706 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.727+0000 7fb3117fa640 1 -- 172.21.15.104:0/2133959057 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb304050260 con 0x7fb3141403f0 2024-03-29T11:20:55.706 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.727+0000 7fb3117fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:55.706 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:55.727+0000 7fb3117fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:56.024 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:56.047+0000 7fb31a4b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:56.024 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:56.047+0000 7fb31a4b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:56.024 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:56.047+0000 7fb31a4b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:20:56.024 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:56.047+0000 7fb31a4b3640 1 -- 172.21.15.104:0/2133959057 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb31413ed30 con 0x7fb3141403f0 2024-03-29T11:20:56.025 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:56.049+0000 7fb3117fa640 1 -- 172.21.15.104:0/2133959057 <== mon.1 v1:172.21.15.119:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7fb304052190 con 0x7fb3141403f0 2024-03-29T11:20:56.026 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:56.049+0000 7fb3117fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:56.026 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:56.049+0000 7fb3117fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:56.026 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:20:56.026 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:56.030 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:56.053+0000 7fb31a4b3640 10 librados: watch_flush enter 2024-03-29T11:20:56.030 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:56.053+0000 7fb31a4b3640 10 librados: watch_flush exit 2024-03-29T11:20:56.030 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:56.053+0000 7fb31a4b3640 20 client.4279.objecter shutdown clearing up homeless session... 2024-03-29T11:20:56.030 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:56.053+0000 7fb31a4b3640 10 client.4279.objecter successfully canceled tick 2024-03-29T11:20:56.030 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:56.053+0000 7fb31a4b3640 1 -- 172.21.15.104:0/2133959057 >> v1:172.21.15.104:6816/3341126660 conn(0x7fb2f406dc90 legacy=0x7fb2f4070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:56.030 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:56.053+0000 7fb31a4b3640 10 monclient: shutdown 2024-03-29T11:20:56.030 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:56.053+0000 7fb31a4b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:56.031 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:56.053+0000 7fb31a4b3640 1 -- 172.21.15.104:0/2133959057 >> v1:172.21.15.119:6789/0 conn(0x7fb3141403f0 legacy=0x7fb3141d5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:56.031 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:56.053+0000 7fb31a4b3640 1 -- 172.21.15.104:0/2133959057 shutdown_connections 2024-03-29T11:20:56.031 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:56.053+0000 7fb31a4b3640 1 -- 172.21.15.104:0/2133959057 >> 172.21.15.104:0/2133959057 conn(0x7fb3140b35e0 msgr2=0x7fb3140b1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:56.031 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:56.053+0000 7fb31a4b3640 1 -- 172.21.15.104:0/2133959057 shutdown_connections 2024-03-29T11:20:56.031 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:56.054+0000 7fb31a4b3640 1 -- 172.21.15.104:0/2133959057 wait complete. 2024-03-29T11:20:56.031 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:20:56.054+0000 7fb31a4b3640 1 librados: shutdown 2024-03-29T11:20:56.047 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:02.854 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:03.004 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.026+0000 7f279fe8f640 10 monclient: get_monmap_and_config 2024-03-29T11:21:03.004 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.026+0000 7f279fe8f640 10 monclient: build_initial_monmap 2024-03-29T11:21:03.004 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.027+0000 7f279fe8f640 10 monclient: monmap: 2024-03-29T11:21:03.004 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:21:03.004 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:21:03.004 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:21:03.029026+0000 2024-03-29T11:21:03.004 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:21:03.029026+0000 2024-03-29T11:21:03.004 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:03.005 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:21:03.005 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:21:03.005 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:21:03.005 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:21:03.005 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:21:03.008 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.029+0000 7f279fe8f640 1 Processor -- start 2024-03-29T11:21:03.009 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.031+0000 7f279fe8f640 1 -- start start 2024-03-29T11:21:03.009 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.031+0000 7f279fe8f640 10 monclient: init 2024-03-29T11:21:03.009 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.032+0000 7f279fe8f640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:03.009 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.032+0000 7f279fe8f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:21:03.009 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.032+0000 7f279fe8f640 10 monclient(hunting): picked mon.noname-b con 0x7f27981457f0 addr v1:172.21.15.104:6790/0 2024-03-29T11:21:03.013 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.035+0000 7f279fe8f640 10 monclient(hunting): picked mon.noname-a con 0x7f279813f650 addr v1:172.21.15.104:6789/0 2024-03-29T11:21:03.013 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.035+0000 7f279fe8f640 10 monclient(hunting): picked mon.noname-c con 0x7f27980a9d70 addr v1:172.21.15.119:6789/0 2024-03-29T11:21:03.013 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.035+0000 7f279fe8f640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27980aaef0 con 0x7f279813f650 2024-03-29T11:21:03.013 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.035+0000 7f279fe8f640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2798145bd0 con 0x7f27980a9d70 2024-03-29T11:21:03.013 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.035+0000 7f279fe8f640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2798145eb0 con 0x7f27981457f0 2024-03-29T11:21:03.013 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.035+0000 7f279fe8f640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:03.013 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.035+0000 7f279fe8f640 10 monclient(hunting): authenticate will time out at 1044.287721s 2024-03-29T11:21:03.013 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.035+0000 7f279d42a640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f279813f650 0x7f27980b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:33631/0 (socket says 172.21.15.104:33631) 2024-03-29T11:21:03.013 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.035+0000 7f279d42a640 1 -- 172.21.15.104:0/1158197704 learned_addr learned my addr 172.21.15.104:0/1158197704 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:21:03.014 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.036+0000 7f279cc29640 1 -- 172.21.15.104:0/1158197704 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 452987562 0 0) 0x7f2798145eb0 con 0x7f27981457f0 2024-03-29T11:21:03.014 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.036+0000 7f279cc29640 10 monclient(hunting): _negotiate 2024-03-29T11:21:03.014 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.036+0000 7f279cc29640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:03.014 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.036+0000 7f279cc29640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:03.014 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.036+0000 7f279cc29640 10 monclient(hunting): my global_id is 4301 2024-03-29T11:21:03.014 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.036+0000 7f279cc29640 1 -- 172.21.15.104:0/1158197704 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2784003520 con 0x7f27981457f0 2024-03-29T11:21:03.014 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.036+0000 7f279cc29640 1 -- 172.21.15.104:0/1158197704 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3748574518 0 0) 0x7f2784003520 con 0x7f27981457f0 2024-03-29T11:21:03.014 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.036+0000 7f279cc29640 1 -- 172.21.15.104:0/1158197704 >> v1:172.21.15.119:6789/0 conn(0x7f27980a9d70 legacy=0x7f279814d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:21:03.018 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.039+0000 7f279cc29640 1 -- 172.21.15.104:0/1158197704 >> v1:172.21.15.104:6789/0 conn(0x7f279813f650 legacy=0x7f27980b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:03.018 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.039+0000 7f279cc29640 10 monclient: _finish_hunting 0 2024-03-29T11:21:03.018 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.039+0000 7f279cc29640 1 monclient: found mon.noname-b 2024-03-29T11:21:03.018 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.039+0000 7f279cc29640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:03.018 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.039+0000 7f279cc29640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.104:6790/0 2024-03-29T11:21:03.018 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.039+0000 7f279cc29640 1 -- 172.21.15.104:0/1158197704 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2798146340 con 0x7f27981457f0 2024-03-29T11:21:03.018 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.039+0000 7f279cc29640 10 monclient: _finish_auth 0 2024-03-29T11:21:03.018 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.039+0000 7f279cc29640 10 monclient: _check_auth_tickets 2024-03-29T11:21:03.019 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.040+0000 7f279cc29640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:03.019 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.040+0000 7f279cc29640 1 -- 172.21.15.104:0/1158197704 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f2794001690 con 0x7f27981457f0 2024-03-29T11:21:03.019 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.040+0000 7f279fe8f640 5 monclient: authenticate success, global_id 4301 2024-03-29T11:21:03.019 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.040+0000 7f279fe8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:03.019 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.040+0000 7f279cc29640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:03.019 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.040+0000 7f279cc29640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:03.019 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.040+0000 7f279cc29640 10 monclient: dump: 2024-03-29T11:21:03.019 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:21:03.019 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:21:03.019 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:03.019 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:03.019 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:03.019 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:21:03.019 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:21:03.019 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:21:03.020 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:21:03.020 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:21:03.020 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.040+0000 7f279cc29640 1 -- 172.21.15.104:0/1158197704 <== mon.2 v1:172.21.15.104:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2794001920 con 0x7f27981457f0 2024-03-29T11:21:03.020 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.040+0000 7f279fe8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:03.020 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.040+0000 7f279cc29640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:03.020 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.040+0000 7f279cc29640 1 -- 172.21.15.104:0/1158197704 <== mon.2 v1:172.21.15.104:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f2794001cb0 con 0x7f27981457f0 2024-03-29T11:21:03.021 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.040+0000 7f279fe8f640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:03.021 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.040+0000 7f279fe8f640 4 set_mon_vals no callback set 2024-03-29T11:21:03.021 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.041+0000 7f279fe8f640 10 monclient: shutdown 2024-03-29T11:21:03.021 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.041+0000 7f279fe8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:03.021 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.041+0000 7f279fe8f640 1 -- 172.21.15.104:0/1158197704 >> v1:172.21.15.104:6790/0 conn(0x7f27981457f0 legacy=0x7f279813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:03.021 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.042+0000 7f279cc29640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:21:03.021 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.043+0000 7f279fe8f640 1 -- 172.21.15.104:0/1158197704 shutdown_connections 2024-03-29T11:21:03.021 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.043+0000 7f279fe8f640 1 -- 172.21.15.104:0/1158197704 >> 172.21.15.104:0/1158197704 conn(0x7f2798066a00 msgr2=0x7f2798066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:03.024 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.043+0000 7f279fe8f640 1 -- 172.21.15.104:0/1158197704 shutdown_connections 2024-03-29T11:21:03.024 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.043+0000 7f279fe8f640 1 -- 172.21.15.104:0/1158197704 wait complete. 2024-03-29T11:21:03.024 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.044+0000 7f279fe8f640 10 monclient: build_initial_monmap 2024-03-29T11:21:03.024 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.044+0000 7f279fe8f640 10 monclient: monmap: 2024-03-29T11:21:03.024 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:21:03.024 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:03.024 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:21:03.024 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:21:03.024 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:03.024 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:21:03.024 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:21:03.024 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:21:03.024 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:21:03.025 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:21:03.025 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.045+0000 7f279fe8f640 1 librados: starting msgr at 2024-03-29T11:21:03.025 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.045+0000 7f279fe8f640 1 librados: starting objecter 2024-03-29T11:21:03.025 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.045+0000 7f279fe8f640 1 Processor -- start 2024-03-29T11:21:03.025 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.045+0000 7f279fe8f640 1 -- start start 2024-03-29T11:21:03.025 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.045+0000 7f279fe8f640 1 librados: setting wanted keys 2024-03-29T11:21:03.025 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.045+0000 7f279fe8f640 1 librados: calling monclient init 2024-03-29T11:21:03.025 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.045+0000 7f279fe8f640 10 monclient: init 2024-03-29T11:21:03.025 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.046+0000 7f279fe8f640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:03.025 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.046+0000 7f279fe8f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:21:03.025 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.046+0000 7f279fe8f640 10 monclient(hunting): picked mon.noname-a con 0x7f27980a9d70 addr v1:172.21.15.104:6789/0 2024-03-29T11:21:03.028 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.047+0000 7f279fe8f640 10 monclient(hunting): picked mon.noname-b con 0x7f279813f650 addr v1:172.21.15.119:6789/0 2024-03-29T11:21:03.028 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.047+0000 7f279fe8f640 10 monclient(hunting): picked mon.noname-c con 0x7f27981457f0 addr v1:172.21.15.104:6790/0 2024-03-29T11:21:03.028 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.047+0000 7f279fe8f640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27981407d0 con 0x7f27980a9d70 2024-03-29T11:21:03.028 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.047+0000 7f279fe8f640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2798143c80 con 0x7f279813f650 2024-03-29T11:21:03.028 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.047+0000 7f279fe8f640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2798143f60 con 0x7f27981457f0 2024-03-29T11:21:03.029 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.047+0000 7f279fe8f640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:03.029 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.047+0000 7f279fe8f640 10 monclient(hunting): authenticate will time out at 1044.299089s 2024-03-29T11:21:03.029 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.050+0000 7f279d42a640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f279813f650 0x7f2798144820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:55035/0 (socket says 172.21.15.104:55035) 2024-03-29T11:21:03.029 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.050+0000 7f279d42a640 1 -- 172.21.15.104:0/153894731 learned_addr learned my addr 172.21.15.104:0/153894731 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:21:03.031 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.053+0000 7f278e7fc640 10 client.?.objecter ms_handle_connect 0x7f27980a9d70 2024-03-29T11:21:03.031 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.053+0000 7f278e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:03.031 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.053+0000 7f278e7fc640 10 client.?.objecter ms_handle_connect 0x7f279813f650 2024-03-29T11:21:03.031 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.053+0000 7f278e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:03.041 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.057+0000 7f278e7fc640 1 -- 172.21.15.104:0/153894731 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3603142981 0 0) 0x7f2798143c80 con 0x7f279813f650 2024-03-29T11:21:03.041 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.057+0000 7f278e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:03.041 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.057+0000 7f278e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:03.041 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.057+0000 7f278e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:03.041 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.058+0000 7f278e7fc640 10 monclient(hunting): my global_id is 4282 2024-03-29T11:21:03.041 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.058+0000 7f278e7fc640 1 -- 172.21.15.104:0/153894731 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f277c004eb0 con 0x7f279813f650 2024-03-29T11:21:03.041 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.058+0000 7f278e7fc640 1 -- 172.21.15.104:0/153894731 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2850695053 0 0) 0x7f27981407d0 con 0x7f27980a9d70 2024-03-29T11:21:03.042 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.058+0000 7f278e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:03.042 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.058+0000 7f278e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:03.042 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.058+0000 7f278e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:03.042 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.058+0000 7f278e7fc640 10 monclient(hunting): my global_id is 4293 2024-03-29T11:21:03.042 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.058+0000 7f278e7fc640 1 -- 172.21.15.104:0/153894731 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2798143c80 con 0x7f27980a9d70 2024-03-29T11:21:03.042 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.058+0000 7f278e7fc640 1 -- 172.21.15.104:0/153894731 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 796117906 0 0) 0x7f277c004eb0 con 0x7f279813f650 2024-03-29T11:21:03.042 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.058+0000 7f278e7fc640 1 -- 172.21.15.104:0/153894731 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27981407d0 con 0x7f279813f650 2024-03-29T11:21:03.042 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.058+0000 7f278e7fc640 1 -- 172.21.15.104:0/153894731 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f2788001510 con 0x7f279813f650 2024-03-29T11:21:03.042 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.058+0000 7f278e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:03.042 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.058+0000 7f278e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:03.042 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.058+0000 7f278e7fc640 10 monclient(hunting): dump: 2024-03-29T11:21:03.042 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:21:03.042 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:21:03.042 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:03.043 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:03.043 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:03.043 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:21:03.043 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:21:03.043 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:21:03.043 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:21:03.043 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:21:03.043 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.058+0000 7f278e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:03.043 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.059+0000 7f278e7fc640 10 client.?.objecter ms_handle_connect 0x7f27981457f0 2024-03-29T11:21:03.043 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.059+0000 7f278e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:03.043 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.059+0000 7f278e7fc640 1 -- 172.21.15.104:0/153894731 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2554000834 0 0) 0x7f2798143f60 con 0x7f27981457f0 2024-03-29T11:21:03.043 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.059+0000 7f278e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:03.043 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.059+0000 7f278e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:03.043 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.059+0000 7f278e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:03.044 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.059+0000 7f278e7fc640 10 monclient(hunting): my global_id is 4304 2024-03-29T11:21:03.044 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.059+0000 7f278e7fc640 1 -- 172.21.15.104:0/153894731 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f277c004eb0 con 0x7f27981457f0 2024-03-29T11:21:03.044 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.059+0000 7f278e7fc640 1 -- 172.21.15.104:0/153894731 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2762393662 0 0) 0x7f277c004eb0 con 0x7f27981457f0 2024-03-29T11:21:03.044 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.060+0000 7f278e7fc640 1 -- 172.21.15.104:0/153894731 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2798143f60 con 0x7f27981457f0 2024-03-29T11:21:03.044 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.060+0000 7f278e7fc640 1 -- 172.21.15.104:0/153894731 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f2790001a70 con 0x7f27981457f0 2024-03-29T11:21:03.044 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.060+0000 7f278e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:03.044 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.060+0000 7f278e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:03.044 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.060+0000 7f278e7fc640 10 monclient(hunting): dump: 2024-03-29T11:21:03.044 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:21:03.044 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:21:03.044 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:03.044 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:03.044 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:03.044 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:21:03.045 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:21:03.045 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:21:03.045 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:21:03.045 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:21:03.045 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.060+0000 7f278e7fc640 1 -- 172.21.15.104:0/153894731 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4242436097 0 0) 0x7f2798143c80 con 0x7f27980a9d70 2024-03-29T11:21:03.045 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.060+0000 7f278e7fc640 1 -- 172.21.15.104:0/153894731 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f277c004eb0 con 0x7f27980a9d70 2024-03-29T11:21:03.045 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.060+0000 7f278e7fc640 1 -- 172.21.15.104:0/153894731 <== mon.2 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1655141230 0 0) 0x7f2798143f60 con 0x7f27981457f0 2024-03-29T11:21:03.045 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.060+0000 7f278e7fc640 1 -- 172.21.15.104:0/153894731 >> v1:172.21.15.119:6789/0 conn(0x7f279813f650 legacy=0x7f2798144820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:03.045 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.060+0000 7f278e7fc640 1 -- 172.21.15.104:0/153894731 >> v1:172.21.15.104:6789/0 conn(0x7f27980a9d70 legacy=0x7f27981d6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:03.045 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.060+0000 7f278e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:21:03.045 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.060+0000 7f278e7fc640 1 monclient: found mon.c 2024-03-29T11:21:03.045 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.060+0000 7f278e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:03.045 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.061+0000 7f278e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:21:03.045 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.061+0000 7f278e7fc640 1 -- 172.21.15.104:0/153894731 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2798140a20 con 0x7f27981457f0 2024-03-29T11:21:03.045 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.061+0000 7f278e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:21:03.046 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.061+0000 7f278e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:21:03.046 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.061+0000 7f278e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:03.046 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.061+0000 7f279fe8f640 5 monclient: authenticate success, global_id 4304 2024-03-29T11:21:03.046 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.061+0000 7f279fe8f640 10 monclient: _renew_subs 2024-03-29T11:21:03.046 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.061+0000 7f279fe8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:21:03.046 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.061+0000 7f279fe8f640 1 -- 172.21.15.104:0/153894731 --> v1:172.21.15.104:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2798071f30 con 0x7f27981457f0 2024-03-29T11:21:03.046 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.061+0000 7f279fe8f640 10 client.4304.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:03.046 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.061+0000 7f279fe8f640 10 monclient: _renew_subs 2024-03-29T11:21:03.046 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.061+0000 7f279fe8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:21:03.046 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.061+0000 7f279fe8f640 1 -- 172.21.15.104:0/153894731 --> v1:172.21.15.104:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f279813ff80 con 0x7f27981457f0 2024-03-29T11:21:03.046 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.061+0000 7f279fe8f640 1 librados: init done 2024-03-29T11:21:03.046 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.061+0000 7f278e7fc640 1 -- 172.21.15.104:0/153894731 <== mon.2 v1:172.21.15.104:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2790001f80 con 0x7f27981457f0 2024-03-29T11:21:03.046 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.061+0000 7f278e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:03.046 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.061+0000 7f278e7fc640 1 -- 172.21.15.104:0/153894731 <== mon.2 v1:172.21.15.104:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f2790003a10 con 0x7f27981457f0 2024-03-29T11:21:03.047 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.061+0000 7f278e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:03.047 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.061+0000 7f278e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:03.047 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.061+0000 7f278e7fc640 10 monclient: dump: 2024-03-29T11:21:03.047 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:21:03.047 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:21:03.047 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:03.047 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:03.047 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:03.047 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:21:03.047 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:21:03.047 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:21:03.047 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:21:03.047 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:21:03.047 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.062+0000 7f278ffff640 4 set_mon_vals no callback set 2024-03-29T11:21:03.047 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.062+0000 7f279fe8f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:03.048 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.063+0000 7f279fe8f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:03.048 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.063+0000 7f279fe8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:21:03.048 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.063+0000 7f279fe8f640 1 -- 172.21.15.104:0/153894731 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2780005080 con 0x7f27981457f0 2024-03-29T11:21:03.049 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.063+0000 7f278e7fc640 1 -- 172.21.15.104:0/153894731 <== mon.2 v1:172.21.15.104:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f27900199e0 con 0x7f27981457f0 2024-03-29T11:21:03.049 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.064+0000 7f278e7fc640 1 -- 172.21.15.104:0/153894731 <== mon.2 v1:172.21.15.104:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f2790083940 con 0x7f27981457f0 2024-03-29T11:21:03.049 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.064+0000 7f278e7fc640 10 client.4304.objecter ms_dispatch 0x7f2798000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:03.049 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.064+0000 7f278e7fc640 3 client.4304.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:03.049 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.064+0000 7f278e7fc640 3 client.4304.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:03.049 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.064+0000 7f278e7fc640 20 client.4304.objecter dump_active .. 0 homeless 2024-03-29T11:21:03.049 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.064+0000 7f279dc2b640 10 monclient: get_auth_request con 0x7f277c06e0d0 auth_method 0 2024-03-29T11:21:03.049 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.064+0000 7f278e7fc640 10 client.4304.objecter ms_handle_connect 0x7f277c06e0d0 2024-03-29T11:21:03.051 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.074+0000 7f278e7fc640 1 -- 172.21.15.104:0/153894731 <== mon.2 v1:172.21.15.104:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f279004f050 con 0x7f27981457f0 2024-03-29T11:21:03.052 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.074+0000 7f278e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:03.052 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.074+0000 7f278e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:03.375 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.397+0000 7f279fe8f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:03.375 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.397+0000 7f279fe8f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:03.375 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.397+0000 7f279fe8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:21:03.375 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.397+0000 7f279fe8f640 1 -- 172.21.15.104:0/153894731 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2780005330 con 0x7f27981457f0 2024-03-29T11:21:03.377 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.399+0000 7f278e7fc640 1 -- 172.21.15.104:0/153894731 <== mon.2 v1:172.21.15.104:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f2790051150 con 0x7f27981457f0 2024-03-29T11:21:03.377 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.399+0000 7f278e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:03.377 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.399+0000 7f278e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:03.379 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:21:03.379 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:03.383 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.405+0000 7f279fe8f640 10 librados: watch_flush enter 2024-03-29T11:21:03.383 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.405+0000 7f279fe8f640 10 librados: watch_flush exit 2024-03-29T11:21:03.383 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.405+0000 7f279fe8f640 20 client.4304.objecter shutdown clearing up homeless session... 2024-03-29T11:21:03.383 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.405+0000 7f279fe8f640 10 client.4304.objecter successfully canceled tick 2024-03-29T11:21:03.383 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.405+0000 7f279fe8f640 1 -- 172.21.15.104:0/153894731 >> v1:172.21.15.104:6816/3341126660 conn(0x7f277c06e0d0 legacy=0x7f277c070590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:03.384 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.405+0000 7f279fe8f640 10 monclient: shutdown 2024-03-29T11:21:03.384 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.405+0000 7f279fe8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:03.384 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.405+0000 7f279fe8f640 1 -- 172.21.15.104:0/153894731 >> v1:172.21.15.104:6790/0 conn(0x7f27981457f0 legacy=0x7f2798142830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:03.384 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.406+0000 7f279fe8f640 1 -- 172.21.15.104:0/153894731 shutdown_connections 2024-03-29T11:21:03.384 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.406+0000 7f279fe8f640 1 -- 172.21.15.104:0/153894731 >> 172.21.15.104:0/153894731 conn(0x7f2798066a00 msgr2=0x7f27981da9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:03.385 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.408+0000 7f279fe8f640 1 -- 172.21.15.104:0/153894731 shutdown_connections 2024-03-29T11:21:03.386 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.408+0000 7f279fe8f640 1 -- 172.21.15.104:0/153894731 wait complete. 2024-03-29T11:21:03.386 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:03.409+0000 7f279fe8f640 1 librados: shutdown 2024-03-29T11:21:03.407 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:10.213 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:10.362 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.385+0000 7f12b4d82640 10 monclient: get_monmap_and_config 2024-03-29T11:21:10.362 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.385+0000 7f12b4d82640 10 monclient: build_initial_monmap 2024-03-29T11:21:10.362 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.385+0000 7f12b4d82640 10 monclient: monmap: 2024-03-29T11:21:10.362 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:21:10.363 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:21:10.363 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:21:10.386241+0000 2024-03-29T11:21:10.363 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:21:10.386241+0000 2024-03-29T11:21:10.363 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:10.363 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:21:10.363 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:21:10.363 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:21:10.363 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:21:10.363 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:21:10.363 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.386+0000 7f12b4d82640 1 Processor -- start 2024-03-29T11:21:10.363 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.386+0000 7f12b4d82640 1 -- start start 2024-03-29T11:21:10.363 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.386+0000 7f12b4d82640 10 monclient: init 2024-03-29T11:21:10.364 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.387+0000 7f12b4d82640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:10.364 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.387+0000 7f12b4d82640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:21:10.364 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.387+0000 7f12b4d82640 10 monclient(hunting): picked mon.noname-a con 0x7f12b00ae250 addr v1:172.21.15.104:6789/0 2024-03-29T11:21:10.364 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.387+0000 7f12b4d82640 10 monclient(hunting): picked mon.noname-c con 0x7f12b013f970 addr v1:172.21.15.119:6789/0 2024-03-29T11:21:10.364 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.387+0000 7f12b4d82640 10 monclient(hunting): picked mon.noname-b con 0x7f12b0142800 addr v1:172.21.15.104:6790/0 2024-03-29T11:21:10.365 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.387+0000 7f12b4d82640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12b00afe00 con 0x7f12b00ae250 2024-03-29T11:21:10.365 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.387+0000 7f12b4d82640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12b0142c10 con 0x7f12b013f970 2024-03-29T11:21:10.365 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.387+0000 7f12b4d82640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12b00afb20 con 0x7f12b0142800 2024-03-29T11:21:10.365 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.387+0000 7f12b4d82640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:10.365 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.387+0000 7f12b4d82640 10 monclient(hunting): authenticate will time out at 1051.638358s 2024-03-29T11:21:10.367 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.390+0000 7f12ae59c640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f12b00ae250 0x7f12b013f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:37041/0 (socket says 172.21.15.104:37041) 2024-03-29T11:21:10.367 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.390+0000 7f12aed9d640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f12b0142800 0x7f12b014da10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:36433/0 (socket says 172.21.15.104:36433) 2024-03-29T11:21:10.367 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.390+0000 7f12ae59c640 1 -- 172.21.15.104:0/3710257243 learned_addr learned my addr 172.21.15.104:0/3710257243 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:21:10.368 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.391+0000 7f12ad59a640 1 -- 172.21.15.104:0/3710257243 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 138471862 0 0) 0x7f12b00afe00 con 0x7f12b00ae250 2024-03-29T11:21:10.368 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.391+0000 7f12ad59a640 10 monclient(hunting): _negotiate 2024-03-29T11:21:10.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.391+0000 7f12ad59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:10.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.391+0000 7f12ad59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:10.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.391+0000 7f12ad59a640 10 monclient(hunting): my global_id is 4296 2024-03-29T11:21:10.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.391+0000 7f12ad59a640 1 -- 172.21.15.104:0/3710257243 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1298003520 con 0x7f12b00ae250 2024-03-29T11:21:10.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.392+0000 7f12ad59a640 1 -- 172.21.15.104:0/3710257243 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 566550168 0 0) 0x7f12b00afb20 con 0x7f12b0142800 2024-03-29T11:21:10.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.392+0000 7f12ad59a640 10 monclient(hunting): _negotiate 2024-03-29T11:21:10.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.392+0000 7f12ad59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:10.377 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.392+0000 7f12ad59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:10.378 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.392+0000 7f12ad59a640 10 monclient(hunting): my global_id is 4307 2024-03-29T11:21:10.378 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.392+0000 7f12ad59a640 1 -- 172.21.15.104:0/3710257243 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12b00afe00 con 0x7f12b0142800 2024-03-29T11:21:10.378 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.392+0000 7f12ad59a640 1 -- 172.21.15.104:0/3710257243 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2514821562 0 0) 0x7f12b0142c10 con 0x7f12b013f970 2024-03-29T11:21:10.378 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.392+0000 7f12ad59a640 10 monclient(hunting): _negotiate 2024-03-29T11:21:10.378 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.392+0000 7f12ad59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:10.378 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.392+0000 7f12ad59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:10.379 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.392+0000 7f12ad59a640 10 monclient(hunting): my global_id is 4285 2024-03-29T11:21:10.379 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.392+0000 7f12ad59a640 1 -- 172.21.15.104:0/3710257243 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12b00afb20 con 0x7f12b013f970 2024-03-29T11:21:10.379 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.392+0000 7f12ad59a640 1 -- 172.21.15.104:0/3710257243 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 214390269 0 0) 0x7f1298003520 con 0x7f12b00ae250 2024-03-29T11:21:10.379 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.392+0000 7f12ad59a640 1 -- 172.21.15.104:0/3710257243 >> v1:172.21.15.104:6790/0 conn(0x7f12b0142800 legacy=0x7f12b014da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:10.379 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.392+0000 7f12ad59a640 1 -- 172.21.15.104:0/3710257243 >> v1:172.21.15.119:6789/0 conn(0x7f12b013f970 legacy=0x7f12b01420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:10.379 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.392+0000 7f12ad59a640 10 monclient: _finish_hunting 0 2024-03-29T11:21:10.380 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.392+0000 7f12ad59a640 1 monclient: found mon.noname-a 2024-03-29T11:21:10.380 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.392+0000 7f12ad59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:10.380 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.392+0000 7f12ad59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.104:6789/0 2024-03-29T11:21:10.380 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.392+0000 7f12ad59a640 1 -- 172.21.15.104:0/3710257243 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b00aab80 con 0x7f12b00ae250 2024-03-29T11:21:10.380 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.392+0000 7f12ad59a640 10 monclient: _finish_auth 0 2024-03-29T11:21:10.380 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.392+0000 7f12ad59a640 10 monclient: _check_auth_tickets 2024-03-29T11:21:10.380 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.392+0000 7f12ad59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:10.380 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.392+0000 7f12ad59a640 1 -- 172.21.15.104:0/3710257243 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f12a0001690 con 0x7f12b00ae250 2024-03-29T11:21:10.381 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.392+0000 7f12ad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:10.381 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.392+0000 7f12ad59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:10.381 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.392+0000 7f12ad59a640 10 monclient: dump: 2024-03-29T11:21:10.381 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:21:10.381 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:21:10.381 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:10.381 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:10.381 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:10.382 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:21:10.382 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:21:10.382 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:21:10.382 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:21:10.382 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:21:10.382 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.395+0000 7f12ad59a640 1 -- 172.21.15.104:0/3710257243 <== mon.0 v1:172.21.15.104:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f12a0001870 con 0x7f12b00ae250 2024-03-29T11:21:10.383 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.395+0000 7f12ad59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:10.383 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.395+0000 7f12ad59a640 1 -- 172.21.15.104:0/3710257243 <== mon.0 v1:172.21.15.104:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f12a0001690 con 0x7f12b00ae250 2024-03-29T11:21:10.383 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.396+0000 7f12ad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:10.383 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.396+0000 7f12ad59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:10.383 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.396+0000 7f12ad59a640 10 monclient: dump: 2024-03-29T11:21:10.383 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:21:10.383 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:21:10.384 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:10.384 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:10.384 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:10.384 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:21:10.384 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:21:10.384 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:21:10.385 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:21:10.385 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:21:10.385 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.396+0000 7f12b4d82640 5 monclient: authenticate success, global_id 4296 2024-03-29T11:21:10.385 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.396+0000 7f12b4d82640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:10.385 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.396+0000 7f12b4d82640 4 set_mon_vals no callback set 2024-03-29T11:21:10.385 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.397+0000 7f12b4d82640 10 monclient: shutdown 2024-03-29T11:21:10.385 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.397+0000 7f12b4d82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:10.386 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.397+0000 7f12b4d82640 1 -- 172.21.15.104:0/3710257243 >> v1:172.21.15.104:6789/0 conn(0x7f12b00ae250 legacy=0x7f12b013f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:10.386 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.399+0000 7f12b4d82640 1 -- 172.21.15.104:0/3710257243 shutdown_connections 2024-03-29T11:21:10.386 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.399+0000 7f12b4d82640 1 -- 172.21.15.104:0/3710257243 >> 172.21.15.104:0/3710257243 conn(0x7f12b00b35e0 msgr2=0x7f12b00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:10.386 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.400+0000 7f12b4d82640 1 -- 172.21.15.104:0/3710257243 shutdown_connections 2024-03-29T11:21:10.398 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.403+0000 7f12b4d82640 1 -- 172.21.15.104:0/3710257243 wait complete. 2024-03-29T11:21:10.398 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.404+0000 7f12b4d82640 10 monclient: build_initial_monmap 2024-03-29T11:21:10.398 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.404+0000 7f12b4d82640 10 monclient: monmap: 2024-03-29T11:21:10.398 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:21:10.398 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:10.398 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:21:10.398 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:21:10.398 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:10.398 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:21:10.398 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:21:10.398 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:21:10.398 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:21:10.399 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:21:10.399 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.405+0000 7f12b4d82640 1 librados: starting msgr at 2024-03-29T11:21:10.399 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.405+0000 7f12b4d82640 1 librados: starting objecter 2024-03-29T11:21:10.399 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.405+0000 7f12b4d82640 1 Processor -- start 2024-03-29T11:21:10.399 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.405+0000 7f12b4d82640 1 -- start start 2024-03-29T11:21:10.399 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.405+0000 7f12b4d82640 1 librados: setting wanted keys 2024-03-29T11:21:10.399 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.405+0000 7f12b4d82640 1 librados: calling monclient init 2024-03-29T11:21:10.399 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.405+0000 7f12b4d82640 10 monclient: init 2024-03-29T11:21:10.399 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.406+0000 7f12b4d82640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:10.399 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.406+0000 7f12b4d82640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:10.399 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.406+0000 7f12b4d82640 10 monclient(hunting): picked mon.noname-c con 0x7f12b00ae250 addr v1:172.21.15.104:6790/0 2024-03-29T11:21:10.399 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.406+0000 7f12b4d82640 10 monclient(hunting): picked mon.noname-b con 0x7f12b013f970 addr v1:172.21.15.119:6789/0 2024-03-29T11:21:10.399 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.406+0000 7f12b4d82640 10 monclient(hunting): picked mon.noname-a con 0x7f12b0142800 addr v1:172.21.15.104:6789/0 2024-03-29T11:21:10.399 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.406+0000 7f12b4d82640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12b013a4c0 con 0x7f12b0142800 2024-03-29T11:21:10.400 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.406+0000 7f12b4d82640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12b0135f20 con 0x7f12b013f970 2024-03-29T11:21:10.400 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.406+0000 7f12b4d82640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12b0136200 con 0x7f12b00ae250 2024-03-29T11:21:10.400 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.406+0000 7f12b4d82640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:10.400 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.406+0000 7f12b4d82640 10 monclient(hunting): authenticate will time out at 1051.657409s 2024-03-29T11:21:10.400 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.409+0000 7f12aed9d640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f12b0142800 0x7f12b01f3510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:37273/0 (socket says 172.21.15.104:37273) 2024-03-29T11:21:10.400 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.409+0000 7f12aed9d640 1 -- 172.21.15.104:0/2077513203 learned_addr learned my addr 172.21.15.104:0/2077513203 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:21:10.400 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.409+0000 7f129effd640 10 client.?.objecter ms_handle_connect 0x7f12b0142800 2024-03-29T11:21:10.400 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.409+0000 7f129effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:10.400 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.409+0000 7f129effd640 1 -- 172.21.15.104:0/2077513203 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 930689877 0 0) 0x7f12b013a4c0 con 0x7f12b0142800 2024-03-29T11:21:10.400 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.410+0000 7f129effd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:10.400 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.410+0000 7f129effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:10.400 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.410+0000 7f129effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:10.400 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.410+0000 7f129effd640 10 monclient(hunting): my global_id is 4299 2024-03-29T11:21:10.400 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.410+0000 7f129effd640 1 -- 172.21.15.104:0/2077513203 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1290004eb0 con 0x7f12b0142800 2024-03-29T11:21:10.400 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.411+0000 7f129effd640 1 -- 172.21.15.104:0/2077513203 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1308331767 0 0) 0x7f1290004eb0 con 0x7f12b0142800 2024-03-29T11:21:10.401 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.411+0000 7f129effd640 1 -- 172.21.15.104:0/2077513203 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f12b013a4c0 con 0x7f12b0142800 2024-03-29T11:21:10.401 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.411+0000 7f129effd640 1 -- 172.21.15.104:0/2077513203 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f12a8001b70 con 0x7f12b0142800 2024-03-29T11:21:10.401 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.411+0000 7f129effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:10.401 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.411+0000 7f129effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:10.401 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.411+0000 7f129effd640 10 monclient(hunting): dump: 2024-03-29T11:21:10.401 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:21:10.401 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:21:10.401 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:10.401 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:10.401 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:10.401 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:21:10.401 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:21:10.401 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:21:10.401 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:21:10.402 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:21:10.402 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.411+0000 7f129effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:10.402 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.412+0000 7f129effd640 1 -- 172.21.15.104:0/2077513203 <== mon.0 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2901844983 0 0) 0x7f12b013a4c0 con 0x7f12b0142800 2024-03-29T11:21:10.402 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.412+0000 7f129effd640 1 -- 172.21.15.104:0/2077513203 >> v1:172.21.15.104:6790/0 conn(0x7f12b00ae250 legacy=0x7f12b00ac2b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:21:10.402 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.413+0000 7f129effd640 1 -- 172.21.15.104:0/2077513203 >> v1:172.21.15.119:6789/0 conn(0x7f12b013f970 legacy=0x7f12b0145600 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:21:10.402 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.413+0000 7f129effd640 10 monclient: _finish_hunting 0 2024-03-29T11:21:10.402 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.413+0000 7f129effd640 1 monclient: found mon.a 2024-03-29T11:21:10.402 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.413+0000 7f129effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:10.402 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.413+0000 7f129effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:21:10.402 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.413+0000 7f129effd640 1 -- 172.21.15.104:0/2077513203 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b0136720 con 0x7f12b0142800 2024-03-29T11:21:10.402 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.413+0000 7f129effd640 10 monclient: _finish_auth 0 2024-03-29T11:21:10.402 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.413+0000 7f129effd640 10 monclient: _check_auth_tickets 2024-03-29T11:21:10.403 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.413+0000 7f129effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:10.403 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.414+0000 7f12b4d82640 5 monclient: authenticate success, global_id 4299 2024-03-29T11:21:10.403 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.414+0000 7f12b4d82640 10 monclient: _renew_subs 2024-03-29T11:21:10.403 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.414+0000 7f12b4d82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:21:10.403 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.414+0000 7f12b4d82640 1 -- 172.21.15.104:0/2077513203 --> v1:172.21.15.104:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12b01462a0 con 0x7f12b0142800 2024-03-29T11:21:10.403 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.414+0000 7f12b4d82640 10 client.4299.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:10.403 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.414+0000 7f12b4d82640 10 monclient: _renew_subs 2024-03-29T11:21:10.403 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.414+0000 7f12b4d82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:21:10.403 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.414+0000 7f12b4d82640 1 -- 172.21.15.104:0/2077513203 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f12b0145230 con 0x7f12b0142800 2024-03-29T11:21:10.403 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.414+0000 7f12b4d82640 1 librados: init done 2024-03-29T11:21:10.403 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.414+0000 7f129effd640 1 -- 172.21.15.104:0/2077513203 <== mon.0 v1:172.21.15.104:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f12a8001d40 con 0x7f12b0142800 2024-03-29T11:21:10.403 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.414+0000 7f129effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:10.404 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.414+0000 7f12acd99640 4 set_mon_vals no callback set 2024-03-29T11:21:10.404 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.415+0000 7f129effd640 1 -- 172.21.15.104:0/2077513203 <== mon.0 v1:172.21.15.104:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f12a8004dc0 con 0x7f12b0142800 2024-03-29T11:21:10.404 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.416+0000 7f129effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:10.404 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.416+0000 7f129effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:10.404 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.416+0000 7f129effd640 10 monclient: dump: 2024-03-29T11:21:10.404 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:21:10.404 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:21:10.404 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:10.404 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:10.404 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:10.404 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:21:10.404 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:21:10.404 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:21:10.405 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:21:10.405 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:21:10.405 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.416+0000 7f129effd640 1 -- 172.21.15.104:0/2077513203 <== mon.0 v1:172.21.15.104:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f12a8004fe0 con 0x7f12b0142800 2024-03-29T11:21:10.405 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.417+0000 7f129effd640 1 -- 172.21.15.104:0/2077513203 <== mon.0 v1:172.21.15.104:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f12a8084da0 con 0x7f12b0142800 2024-03-29T11:21:10.405 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.417+0000 7f129effd640 10 client.4299.objecter ms_dispatch 0x7f12b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:10.405 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.417+0000 7f129effd640 3 client.4299.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:10.405 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.417+0000 7f129effd640 3 client.4299.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:10.405 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.417+0000 7f12b4d82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:10.405 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.417+0000 7f12b4d82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:10.405 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.417+0000 7f12b4d82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:21:10.405 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.417+0000 7f12b4d82640 1 -- 172.21.15.104:0/2077513203 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1294005080 con 0x7f12b0142800 2024-03-29T11:21:10.405 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.417+0000 7f129effd640 20 client.4299.objecter dump_active .. 0 homeless 2024-03-29T11:21:10.405 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.417+0000 7f12ae59c640 10 monclient: get_auth_request con 0x7f129006dc20 auth_method 0 2024-03-29T11:21:10.405 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.418+0000 7f129effd640 10 client.4299.objecter ms_handle_connect 0x7f129006dc20 2024-03-29T11:21:10.406 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.429+0000 7f129effd640 1 -- 172.21.15.104:0/2077513203 <== mon.0 v1:172.21.15.104:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f12a8050680 con 0x7f12b0142800 2024-03-29T11:21:10.406 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.429+0000 7f129effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:10.406 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.429+0000 7f129effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:10.730 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.754+0000 7f12b4d82640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:10.730 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.754+0000 7f12b4d82640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:10.730 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.754+0000 7f12b4d82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:21:10.731 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.754+0000 7f12b4d82640 1 -- 172.21.15.104:0/2077513203 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1294005330 con 0x7f12b0142800 2024-03-29T11:21:10.733 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.757+0000 7f129effd640 1 -- 172.21.15.104:0/2077513203 <== mon.0 v1:172.21.15.104:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f12a80525b0 con 0x7f12b0142800 2024-03-29T11:21:10.733 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.757+0000 7f129effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:10.733 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.757+0000 7f129effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:10.734 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:21:10.734 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:10.739 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.762+0000 7f12b4d82640 10 librados: watch_flush enter 2024-03-29T11:21:10.739 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.763+0000 7f12b4d82640 10 librados: watch_flush exit 2024-03-29T11:21:10.739 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.763+0000 7f12b4d82640 20 client.4299.objecter shutdown clearing up homeless session... 2024-03-29T11:21:10.739 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.763+0000 7f12b4d82640 10 client.4299.objecter successfully canceled tick 2024-03-29T11:21:10.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.763+0000 7f12b4d82640 1 -- 172.21.15.104:0/2077513203 >> v1:172.21.15.104:6816/3341126660 conn(0x7f129006dc20 legacy=0x7f12900700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:10.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.763+0000 7f12b4d82640 10 monclient: shutdown 2024-03-29T11:21:10.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.763+0000 7f12b4d82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:10.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.763+0000 7f12b4d82640 1 -- 172.21.15.104:0/2077513203 >> v1:172.21.15.104:6789/0 conn(0x7f12b0142800 legacy=0x7f12b01f3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:10.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.763+0000 7f12b4d82640 1 -- 172.21.15.104:0/2077513203 shutdown_connections 2024-03-29T11:21:10.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.763+0000 7f12b4d82640 1 -- 172.21.15.104:0/2077513203 >> 172.21.15.104:0/2077513203 conn(0x7f12b00b35e0 msgr2=0x7f12b00b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:10.743 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.766+0000 7f12b4d82640 1 -- 172.21.15.104:0/2077513203 shutdown_connections 2024-03-29T11:21:10.743 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.767+0000 7f12b4d82640 1 -- 172.21.15.104:0/2077513203 wait complete. 2024-03-29T11:21:10.743 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:10.767+0000 7f12b4d82640 1 librados: shutdown 2024-03-29T11:21:10.763 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:17.570 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:17.727 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.749+0000 7f87419ed640 10 monclient: get_monmap_and_config 2024-03-29T11:21:17.727 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.749+0000 7f87419ed640 10 monclient: build_initial_monmap 2024-03-29T11:21:17.727 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.749+0000 7f87419ed640 10 monclient: monmap: 2024-03-29T11:21:17.727 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:21:17.727 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:21:17.727 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:21:17.750397+0000 2024-03-29T11:21:17.727 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:21:17.750397+0000 2024-03-29T11:21:17.728 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:17.728 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:21:17.728 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:21:17.728 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:21:17.728 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:21:17.728 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:21:17.728 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.751+0000 7f87419ed640 1 Processor -- start 2024-03-29T11:21:17.728 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.752+0000 7f87419ed640 1 -- start start 2024-03-29T11:21:17.729 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.752+0000 7f87419ed640 10 monclient: init 2024-03-29T11:21:17.732 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.752+0000 7f87419ed640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:17.732 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.752+0000 7f87419ed640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:21:17.732 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.752+0000 7f87419ed640 10 monclient(hunting): picked mon.noname-c con 0x7f873c0b4100 addr v1:172.21.15.119:6789/0 2024-03-29T11:21:17.732 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.752+0000 7f87419ed640 10 monclient(hunting): picked mon.noname-a con 0x7f873c13ee90 addr v1:172.21.15.104:6789/0 2024-03-29T11:21:17.732 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.752+0000 7f87419ed640 10 monclient(hunting): picked mon.noname-b con 0x7f873c0af900 addr v1:172.21.15.104:6790/0 2024-03-29T11:21:17.732 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.752+0000 7f87419ed640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f873c0afd10 con 0x7f873c13ee90 2024-03-29T11:21:17.732 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.753+0000 7f87419ed640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f873c0af6f0 con 0x7f873c0b4100 2024-03-29T11:21:17.732 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.753+0000 7f87419ed640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f873c0aac80 con 0x7f873c0af900 2024-03-29T11:21:17.733 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.753+0000 7f87419ed640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:17.733 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.753+0000 7f87419ed640 10 monclient(hunting): authenticate will time out at 1059.004040s 2024-03-29T11:21:17.733 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.756+0000 7f873b7fe640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f873c0af900 0x7f873c14be10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:38625/0 (socket says 172.21.15.104:38625) 2024-03-29T11:21:17.733 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.756+0000 7f873b7fe640 1 -- 172.21.15.104:0/534329523 learned_addr learned my addr 172.21.15.104:0/534329523 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:21:17.733 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.757+0000 7f8739ffb640 1 -- 172.21.15.104:0/534329523 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1333546770 0 0) 0x7f873c0afd10 con 0x7f873c13ee90 2024-03-29T11:21:17.733 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.757+0000 7f8739ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:17.733 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.757+0000 7f8739ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:17.733 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.757+0000 7f8739ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:17.733 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.757+0000 7f8739ffb640 10 monclient(hunting): my global_id is 4302 2024-03-29T11:21:17.733 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.757+0000 7f8739ffb640 1 -- 172.21.15.104:0/534329523 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8724003520 con 0x7f873c13ee90 2024-03-29T11:21:17.736 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.760+0000 7f8739ffb640 1 -- 172.21.15.104:0/534329523 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3327505210 0 0) 0x7f8724003520 con 0x7f873c13ee90 2024-03-29T11:21:17.736 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.760+0000 7f8739ffb640 1 -- 172.21.15.104:0/534329523 >> v1:172.21.15.104:6790/0 conn(0x7f873c0af900 legacy=0x7f873c14be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:17.736 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.760+0000 7f8739ffb640 1 -- 172.21.15.104:0/534329523 >> v1:172.21.15.119:6789/0 conn(0x7f873c0b4100 legacy=0x7f873c0adfe0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:21:17.737 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.760+0000 7f8739ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:21:17.737 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.760+0000 7f8739ffb640 1 monclient: found mon.noname-a 2024-03-29T11:21:17.737 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.760+0000 7f8739ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:17.737 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.760+0000 7f8739ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.104:6789/0 2024-03-29T11:21:17.737 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.760+0000 7f8739ffb640 1 -- 172.21.15.104:0/534329523 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f873c0aaf60 con 0x7f873c13ee90 2024-03-29T11:21:17.737 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.760+0000 7f8739ffb640 10 monclient: _finish_auth 0 2024-03-29T11:21:17.737 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.760+0000 7f8739ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:21:17.737 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.760+0000 7f8739ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:17.737 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.760+0000 7f87419ed640 5 monclient: authenticate success, global_id 4302 2024-03-29T11:21:17.737 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.760+0000 7f87419ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:17.737 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.760+0000 7f8739ffb640 1 -- 172.21.15.104:0/534329523 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f8734001a10 con 0x7f873c13ee90 2024-03-29T11:21:17.737 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.760+0000 7f8739ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:17.737 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.760+0000 7f8739ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:17.738 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.760+0000 7f8739ffb640 10 monclient: dump: 2024-03-29T11:21:17.738 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:21:17.738 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:21:17.738 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:17.738 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:17.738 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:17.738 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:21:17.738 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:21:17.738 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:21:17.738 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:21:17.738 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:21:17.738 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.760+0000 7f87419ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:17.738 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.760+0000 7f8739ffb640 1 -- 172.21.15.104:0/534329523 <== mon.0 v1:172.21.15.104:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8734001ca0 con 0x7f873c13ee90 2024-03-29T11:21:17.738 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.760+0000 7f8739ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:17.739 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.760+0000 7f87419ed640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:17.739 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.760+0000 7f87419ed640 4 set_mon_vals no callback set 2024-03-29T11:21:17.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.761+0000 7f8739ffb640 1 -- 172.21.15.104:0/534329523 <== mon.0 v1:172.21.15.104:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f8734001a10 con 0x7f873c13ee90 2024-03-29T11:21:17.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.761+0000 7f8739ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:17.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.761+0000 7f8739ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:17.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.761+0000 7f8739ffb640 10 monclient: dump: 2024-03-29T11:21:17.740 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:21:17.740 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:21:17.740 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:17.740 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:17.740 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:17.740 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:21:17.740 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:21:17.740 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:21:17.740 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:21:17.741 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:21:17.741 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.761+0000 7f87419ed640 10 monclient: shutdown 2024-03-29T11:21:17.741 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.761+0000 7f87419ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:17.741 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.761+0000 7f87419ed640 1 -- 172.21.15.104:0/534329523 >> v1:172.21.15.104:6789/0 conn(0x7f873c13ee90 legacy=0x7f873c13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:17.741 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.761+0000 7f87419ed640 1 -- 172.21.15.104:0/534329523 shutdown_connections 2024-03-29T11:21:17.741 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.761+0000 7f87419ed640 1 -- 172.21.15.104:0/534329523 >> 172.21.15.104:0/534329523 conn(0x7f873c0b35e0 msgr2=0x7f873c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:17.741 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.762+0000 7f87419ed640 1 -- 172.21.15.104:0/534329523 shutdown_connections 2024-03-29T11:21:17.742 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.762+0000 7f87419ed640 1 -- 172.21.15.104:0/534329523 wait complete. 2024-03-29T11:21:17.742 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.763+0000 7f87419ed640 10 monclient: build_initial_monmap 2024-03-29T11:21:17.742 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.763+0000 7f87419ed640 10 monclient: monmap: 2024-03-29T11:21:17.742 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:21:17.743 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:17.743 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:21:17.743 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:21:17.743 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:17.743 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:21:17.743 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:21:17.743 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:21:17.743 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:21:17.743 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:21:17.743 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.764+0000 7f87419ed640 1 librados: starting msgr at 2024-03-29T11:21:17.743 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.764+0000 7f87419ed640 1 librados: starting objecter 2024-03-29T11:21:17.743 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.764+0000 7f87419ed640 1 Processor -- start 2024-03-29T11:21:17.743 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.764+0000 7f87419ed640 1 -- start start 2024-03-29T11:21:17.743 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.764+0000 7f87419ed640 1 librados: setting wanted keys 2024-03-29T11:21:17.743 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.764+0000 7f87419ed640 1 librados: calling monclient init 2024-03-29T11:21:17.743 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.764+0000 7f87419ed640 10 monclient: init 2024-03-29T11:21:17.743 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.765+0000 7f87419ed640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:17.744 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.765+0000 7f87419ed640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:21:17.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.769+0000 7f873affd640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f873c0af900 0x7f873c1432c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:38749/0 (socket says 172.21.15.104:38749) 2024-03-29T11:21:17.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.769+0000 7f873affd640 1 -- 172.21.15.104:0/4023359528 learned_addr learned my addr 172.21.15.104:0/4023359528 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:21:17.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.769+0000 7f87419ed640 10 monclient(hunting): picked mon.noname-c con 0x7f873c0af900 addr v1:172.21.15.104:6790/0 2024-03-29T11:21:17.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.769+0000 7f87419ed640 10 monclient(hunting): picked mon.noname-a con 0x7f873c13ee90 addr v1:172.21.15.104:6789/0 2024-03-29T11:21:17.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.769+0000 7f87419ed640 10 monclient(hunting): picked mon.noname-b con 0x7f873c1daf60 addr v1:172.21.15.119:6789/0 2024-03-29T11:21:17.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.769+0000 7f87419ed640 1 -- 172.21.15.104:0/4023359528 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f873c1307d0 con 0x7f873c13ee90 2024-03-29T11:21:17.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.769+0000 7f87419ed640 1 -- 172.21.15.104:0/4023359528 --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f873c142b30 con 0x7f873c1daf60 2024-03-29T11:21:17.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.769+0000 7f87419ed640 1 -- 172.21.15.104:0/4023359528 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f873c130a20 con 0x7f873c0af900 2024-03-29T11:21:17.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.769+0000 7f87419ed640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:17.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.769+0000 7f87419ed640 10 monclient(hunting): authenticate will time out at 1059.020547s 2024-03-29T11:21:17.747 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.770+0000 7f8723fff640 10 client.?.objecter ms_handle_connect 0x7f873c0af900 2024-03-29T11:21:17.747 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.770+0000 7f8723fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:17.747 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.770+0000 7f8723fff640 1 -- 172.21.15.104:0/4023359528 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3724992453 0 0) 0x7f873c130a20 con 0x7f873c0af900 2024-03-29T11:21:17.747 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.771+0000 7f8723fff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:17.747 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.771+0000 7f8723fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:17.747 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.771+0000 7f8723fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:17.747 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.771+0000 7f8723fff640 10 monclient(hunting): my global_id is 4313 2024-03-29T11:21:17.747 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.771+0000 7f8723fff640 1 -- 172.21.15.104:0/4023359528 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8728004eb0 con 0x7f873c0af900 2024-03-29T11:21:17.749 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.773+0000 7f8723fff640 1 -- 172.21.15.104:0/4023359528 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1987162511 0 0) 0x7f8728004eb0 con 0x7f873c0af900 2024-03-29T11:21:17.749 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.773+0000 7f8723fff640 1 -- 172.21.15.104:0/4023359528 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f873c130a20 con 0x7f873c0af900 2024-03-29T11:21:17.749 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.773+0000 7f8723fff640 1 -- 172.21.15.104:0/4023359528 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f872c001870 con 0x7f873c0af900 2024-03-29T11:21:17.749 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.773+0000 7f8723fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:17.749 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.773+0000 7f8723fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:17.749 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.773+0000 7f8723fff640 10 monclient(hunting): dump: 2024-03-29T11:21:17.749 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:21:17.749 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:21:17.750 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:17.750 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:17.750 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:17.750 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:21:17.750 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:21:17.750 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:21:17.750 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:21:17.750 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:21:17.750 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.773+0000 7f8723fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:17.751 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.774+0000 7f8723fff640 10 client.?.objecter ms_handle_connect 0x7f873c13ee90 2024-03-29T11:21:17.751 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.774+0000 7f8723fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:17.751 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.774+0000 7f8723fff640 10 client.?.objecter ms_handle_connect 0x7f873c1daf60 2024-03-29T11:21:17.751 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.774+0000 7f8723fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:17.751 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.774+0000 7f8723fff640 1 -- 172.21.15.104:0/4023359528 <== mon.2 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2341971564 0 0) 0x7f873c130a20 con 0x7f873c0af900 2024-03-29T11:21:17.751 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.774+0000 7f8723fff640 1 -- 172.21.15.104:0/4023359528 >> v1:172.21.15.119:6789/0 conn(0x7f873c1daf60 legacy=0x7f873c12c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:17.751 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.774+0000 7f8723fff640 1 -- 172.21.15.104:0/4023359528 >> v1:172.21.15.104:6789/0 conn(0x7f873c13ee90 legacy=0x7f873c130f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:17.751 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.774+0000 7f8723fff640 10 monclient: _finish_hunting 0 2024-03-29T11:21:17.752 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.774+0000 7f8723fff640 1 monclient: found mon.c 2024-03-29T11:21:17.752 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.774+0000 7f8723fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:17.752 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.774+0000 7f8723fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:21:17.752 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.774+0000 7f8723fff640 1 -- 172.21.15.104:0/4023359528 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f873c131650 con 0x7f873c0af900 2024-03-29T11:21:17.752 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.774+0000 7f8723fff640 10 monclient: _finish_auth 0 2024-03-29T11:21:17.752 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.774+0000 7f8723fff640 10 monclient: _check_auth_tickets 2024-03-29T11:21:17.752 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.774+0000 7f8723fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:17.752 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.774+0000 7f87419ed640 5 monclient: authenticate success, global_id 4313 2024-03-29T11:21:17.752 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.774+0000 7f87419ed640 10 monclient: _renew_subs 2024-03-29T11:21:17.752 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.774+0000 7f87419ed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:21:17.752 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.774+0000 7f87419ed640 1 -- 172.21.15.104:0/4023359528 --> v1:172.21.15.104:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f873c1404d0 con 0x7f873c0af900 2024-03-29T11:21:17.752 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.774+0000 7f87419ed640 10 client.4313.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:17.752 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.774+0000 7f87419ed640 10 monclient: _renew_subs 2024-03-29T11:21:17.752 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.774+0000 7f87419ed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:21:17.752 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.774+0000 7f87419ed640 1 -- 172.21.15.104:0/4023359528 --> v1:172.21.15.104:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f873c12ddf0 con 0x7f873c0af900 2024-03-29T11:21:17.752 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.774+0000 7f87419ed640 1 librados: init done 2024-03-29T11:21:17.755 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.777+0000 7f87419ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:17.755 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.777+0000 7f87419ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:17.755 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.777+0000 7f87419ed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:21:17.755 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.777+0000 7f87419ed640 1 -- 172.21.15.104:0/4023359528 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f873c130bb0 con 0x7f873c0af900 2024-03-29T11:21:17.757 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.779+0000 7f8723fff640 1 -- 172.21.15.104:0/4023359528 <== mon.2 v1:172.21.15.104:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f872c0019d0 con 0x7f873c0af900 2024-03-29T11:21:17.758 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.779+0000 7f8723fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:17.758 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.779+0000 7f8723fff640 1 -- 172.21.15.104:0/4023359528 <== mon.2 v1:172.21.15.104:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f872c004ab0 con 0x7f873c0af900 2024-03-29T11:21:17.758 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.779+0000 7f8723fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:17.758 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.779+0000 7f8723fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:17.758 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.779+0000 7f8723fff640 10 monclient: dump: 2024-03-29T11:21:17.758 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:21:17.758 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:21:17.758 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:17.758 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:17.758 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:17.758 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:21:17.758 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:21:17.758 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:21:17.759 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:21:17.759 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:21:17.760 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.779+0000 7f87397fa640 4 set_mon_vals no callback set 2024-03-29T11:21:17.760 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.781+0000 7f8723fff640 1 -- 172.21.15.104:0/4023359528 <== mon.2 v1:172.21.15.104:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f872c01ab40 con 0x7f873c0af900 2024-03-29T11:21:17.760 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.782+0000 7f8723fff640 1 -- 172.21.15.104:0/4023359528 <== mon.2 v1:172.21.15.104:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f872c084a10 con 0x7f873c0af900 2024-03-29T11:21:17.760 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.782+0000 7f8723fff640 10 client.4313.objecter ms_dispatch 0x7f873c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:17.760 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.782+0000 7f8723fff640 3 client.4313.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:17.760 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.782+0000 7f8723fff640 3 client.4313.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:17.760 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.782+0000 7f8723fff640 20 client.4313.objecter dump_active .. 0 homeless 2024-03-29T11:21:17.760 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.782+0000 7f873a7fc640 10 monclient: get_auth_request con 0x7f872806dd00 auth_method 0 2024-03-29T11:21:17.760 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.783+0000 7f8723fff640 10 client.4313.objecter ms_handle_connect 0x7f872806dd00 2024-03-29T11:21:17.764 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.788+0000 7f8723fff640 1 -- 172.21.15.104:0/4023359528 <== mon.2 v1:172.21.15.104:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f872c0503b0 con 0x7f873c0af900 2024-03-29T11:21:17.765 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.788+0000 7f8723fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:17.765 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:17.788+0000 7f8723fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:18.094 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:18.117+0000 7f87419ed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:18.094 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:18.117+0000 7f87419ed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:18.094 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:18.117+0000 7f87419ed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:21:18.094 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:18.117+0000 7f87419ed640 1 -- 172.21.15.104:0/4023359528 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f873c06dfd0 con 0x7f873c0af900 2024-03-29T11:21:18.095 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:18.118+0000 7f8723fff640 1 -- 172.21.15.104:0/4023359528 <== mon.2 v1:172.21.15.104:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f872c0522e0 con 0x7f873c0af900 2024-03-29T11:21:18.095 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:18.118+0000 7f8723fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:18.095 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:18.118+0000 7f8723fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:18.095 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:21:18.095 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:18.099 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:18.121+0000 7f87419ed640 10 librados: watch_flush enter 2024-03-29T11:21:18.099 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:18.121+0000 7f87419ed640 10 librados: watch_flush exit 2024-03-29T11:21:18.099 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:18.121+0000 7f87419ed640 20 client.4313.objecter shutdown clearing up homeless session... 2024-03-29T11:21:18.099 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:18.121+0000 7f87419ed640 10 client.4313.objecter successfully canceled tick 2024-03-29T11:21:18.101 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:18.124+0000 7f87419ed640 1 -- 172.21.15.104:0/4023359528 >> v1:172.21.15.104:6816/3341126660 conn(0x7f872806dd00 legacy=0x7f87280701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:18.101 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:18.124+0000 7f87419ed640 10 monclient: shutdown 2024-03-29T11:21:18.101 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:18.124+0000 7f87419ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:18.101 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:18.124+0000 7f87419ed640 1 -- 172.21.15.104:0/4023359528 >> v1:172.21.15.104:6790/0 conn(0x7f873c0af900 legacy=0x7f873c1432c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:18.101 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:18.124+0000 7f87419ed640 1 -- 172.21.15.104:0/4023359528 shutdown_connections 2024-03-29T11:21:18.101 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:18.124+0000 7f87419ed640 1 -- 172.21.15.104:0/4023359528 >> 172.21.15.104:0/4023359528 conn(0x7f873c0b35e0 msgr2=0x7f873c0ad160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:18.101 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:18.125+0000 7f87419ed640 1 -- 172.21.15.104:0/4023359528 shutdown_connections 2024-03-29T11:21:18.101 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:18.125+0000 7f87419ed640 1 -- 172.21.15.104:0/4023359528 wait complete. 2024-03-29T11:21:18.102 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:18.125+0000 7f87419ed640 1 librados: shutdown 2024-03-29T11:21:18.118 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:24.926 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:25.085 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.107+0000 7ff6e5e02640 10 monclient: get_monmap_and_config 2024-03-29T11:21:25.085 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.107+0000 7ff6e5e02640 10 monclient: build_initial_monmap 2024-03-29T11:21:25.085 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.107+0000 7ff6e5e02640 10 monclient: monmap: 2024-03-29T11:21:25.085 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:21:25.085 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:21:25.086 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:21:25.107961+0000 2024-03-29T11:21:25.086 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:21:25.107961+0000 2024-03-29T11:21:25.086 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:25.086 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:21:25.086 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:21:25.086 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:21:25.086 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:21:25.086 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:21:25.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.109+0000 7ff6e5e02640 1 Processor -- start 2024-03-29T11:21:25.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.109+0000 7ff6e5e02640 1 -- start start 2024-03-29T11:21:25.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.109+0000 7ff6e5e02640 10 monclient: init 2024-03-29T11:21:25.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.110+0000 7ff6e5e02640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:25.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.110+0000 7ff6e5e02640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:25.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.110+0000 7ff6e5e02640 10 monclient(hunting): picked mon.noname-b con 0x7ff6e00a9930 addr v1:172.21.15.104:6790/0 2024-03-29T11:21:25.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.110+0000 7ff6e5e02640 10 monclient(hunting): picked mon.noname-c con 0x7ff6e0149dc0 addr v1:172.21.15.119:6789/0 2024-03-29T11:21:25.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.110+0000 7ff6e5e02640 10 monclient(hunting): picked mon.noname-a con 0x7ff6e014ff40 addr v1:172.21.15.104:6789/0 2024-03-29T11:21:25.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.110+0000 7ff6e5e02640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e00aae60 con 0x7ff6e014ff40 2024-03-29T11:21:25.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.110+0000 7ff6e5e02640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e00b4310 con 0x7ff6e0149dc0 2024-03-29T11:21:25.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.110+0000 7ff6e5e02640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e00afdd0 con 0x7ff6e00a9930 2024-03-29T11:21:25.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.110+0000 7ff6e5e02640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:25.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.110+0000 7ff6e5e02640 10 monclient(hunting): authenticate will time out at 1066.361802s 2024-03-29T11:21:25.093 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.117+0000 7ff6dffff640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7ff6e014ff40 0x7ff6e0152330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59169/0 (socket says 172.21.15.104:59169) 2024-03-29T11:21:25.093 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.117+0000 7ff6dffff640 1 -- 172.21.15.104:0/3743013292 learned_addr learned my addr 172.21.15.104:0/3743013292 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:21:25.096 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.118+0000 7ff6de7fc640 1 -- 172.21.15.104:0/3743013292 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2310606101 0 0) 0x7ff6e00b4310 con 0x7ff6e0149dc0 2024-03-29T11:21:25.097 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.118+0000 7ff6de7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:25.097 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.118+0000 7ff6de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:25.097 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.118+0000 7ff6de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:25.097 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.118+0000 7ff6de7fc640 10 monclient(hunting): my global_id is 4288 2024-03-29T11:21:25.097 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.118+0000 7ff6de7fc640 1 -- 172.21.15.104:0/3743013292 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6c8003520 con 0x7ff6e0149dc0 2024-03-29T11:21:25.097 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.118+0000 7ff6de7fc640 1 -- 172.21.15.104:0/3743013292 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3417004605 0 0) 0x7ff6e00aae60 con 0x7ff6e014ff40 2024-03-29T11:21:25.097 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.118+0000 7ff6de7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:25.097 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.118+0000 7ff6de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:25.097 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.118+0000 7ff6de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:25.097 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.118+0000 7ff6de7fc640 10 monclient(hunting): my global_id is 4308 2024-03-29T11:21:25.097 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.118+0000 7ff6de7fc640 1 -- 172.21.15.104:0/3743013292 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6e00b4310 con 0x7ff6e014ff40 2024-03-29T11:21:25.097 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.118+0000 7ff6de7fc640 1 -- 172.21.15.104:0/3743013292 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2822758539 0 0) 0x7ff6e00b4310 con 0x7ff6e014ff40 2024-03-29T11:21:25.097 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.118+0000 7ff6de7fc640 1 -- 172.21.15.104:0/3743013292 >> v1:172.21.15.104:6790/0 conn(0x7ff6e00a9930 legacy=0x7ff6e0143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:25.098 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.118+0000 7ff6de7fc640 1 -- 172.21.15.104:0/3743013292 >> v1:172.21.15.119:6789/0 conn(0x7ff6e0149dc0 legacy=0x7ff6e014a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:25.098 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.118+0000 7ff6de7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:21:25.098 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.118+0000 7ff6de7fc640 1 monclient: found mon.noname-a 2024-03-29T11:21:25.098 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.118+0000 7ff6de7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:25.098 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.118+0000 7ff6de7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.104:6789/0 2024-03-29T11:21:25.098 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.118+0000 7ff6de7fc640 1 -- 172.21.15.104:0/3743013292 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e014a950 con 0x7ff6e014ff40 2024-03-29T11:21:25.098 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.118+0000 7ff6de7fc640 10 monclient: _finish_auth 0 2024-03-29T11:21:25.098 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.118+0000 7ff6de7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:21:25.098 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.118+0000 7ff6de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:25.098 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.119+0000 7ff6e5e02640 5 monclient: authenticate success, global_id 4308 2024-03-29T11:21:25.098 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.119+0000 7ff6e5e02640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:25.098 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.119+0000 7ff6de7fc640 1 -- 172.21.15.104:0/3743013292 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7ff6d8001920 con 0x7ff6e014ff40 2024-03-29T11:21:25.098 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.119+0000 7ff6de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:25.098 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.119+0000 7ff6de7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:25.099 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.119+0000 7ff6de7fc640 10 monclient: dump: 2024-03-29T11:21:25.099 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:21:25.099 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:21:25.099 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:25.099 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:25.099 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:25.099 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:21:25.099 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:21:25.099 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:21:25.099 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:21:25.099 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:21:25.099 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.119+0000 7ff6e5e02640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:25.100 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.121+0000 7ff6de7fc640 1 -- 172.21.15.104:0/3743013292 <== mon.0 v1:172.21.15.104:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6d8001b40 con 0x7ff6e014ff40 2024-03-29T11:21:25.100 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.121+0000 7ff6de7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:25.101 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.121+0000 7ff6e5e02640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:25.101 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.121+0000 7ff6e5e02640 4 set_mon_vals no callback set 2024-03-29T11:21:25.101 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.122+0000 7ff6e5e02640 10 monclient: shutdown 2024-03-29T11:21:25.101 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.122+0000 7ff6e5e02640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:25.101 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.122+0000 7ff6e5e02640 1 -- 172.21.15.104:0/3743013292 >> v1:172.21.15.104:6789/0 conn(0x7ff6e014ff40 legacy=0x7ff6e0152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:25.101 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.123+0000 7ff6e5e02640 1 -- 172.21.15.104:0/3743013292 shutdown_connections 2024-03-29T11:21:25.101 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.123+0000 7ff6e5e02640 1 -- 172.21.15.104:0/3743013292 >> 172.21.15.104:0/3743013292 conn(0x7ff6e00b35e0 msgr2=0x7ff6e00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:25.101 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.123+0000 7ff6e5e02640 1 -- 172.21.15.104:0/3743013292 shutdown_connections 2024-03-29T11:21:25.101 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.124+0000 7ff6e5e02640 1 -- 172.21.15.104:0/3743013292 wait complete. 2024-03-29T11:21:25.102 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.124+0000 7ff6e5e02640 10 monclient: build_initial_monmap 2024-03-29T11:21:25.102 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.124+0000 7ff6e5e02640 10 monclient: monmap: 2024-03-29T11:21:25.102 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:21:25.102 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:25.102 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:21:25.103 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:21:25.103 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:25.103 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:21:25.103 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:21:25.103 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:21:25.103 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:21:25.103 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:21:25.103 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.125+0000 7ff6e5e02640 1 librados: starting msgr at 2024-03-29T11:21:25.103 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.125+0000 7ff6e5e02640 1 librados: starting objecter 2024-03-29T11:21:25.103 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.125+0000 7ff6e5e02640 1 Processor -- start 2024-03-29T11:21:25.103 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.125+0000 7ff6e5e02640 1 -- start start 2024-03-29T11:21:25.103 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.125+0000 7ff6e5e02640 1 librados: setting wanted keys 2024-03-29T11:21:25.103 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.125+0000 7ff6e5e02640 1 librados: calling monclient init 2024-03-29T11:21:25.103 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.125+0000 7ff6e5e02640 10 monclient: init 2024-03-29T11:21:25.105 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.126+0000 7ff6e5e02640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:25.105 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.126+0000 7ff6e5e02640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:21:25.105 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.126+0000 7ff6e5e02640 10 monclient(hunting): picked mon.noname-b con 0x7ff6e00a9930 addr v1:172.21.15.119:6789/0 2024-03-29T11:21:25.105 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.126+0000 7ff6e5e02640 10 monclient(hunting): picked mon.noname-c con 0x7ff6e0149dc0 addr v1:172.21.15.104:6790/0 2024-03-29T11:21:25.106 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.126+0000 7ff6e5e02640 10 monclient(hunting): picked mon.noname-a con 0x7ff6e014ff40 addr v1:172.21.15.104:6789/0 2024-03-29T11:21:25.106 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.126+0000 7ff6e5e02640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e0144a30 con 0x7ff6e014ff40 2024-03-29T11:21:25.106 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.126+0000 7ff6e5e02640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e013bd30 con 0x7ff6e00a9930 2024-03-29T11:21:25.106 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.126+0000 7ff6e5e02640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e00ac7f0 con 0x7ff6e0149dc0 2024-03-29T11:21:25.106 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.126+0000 7ff6e5e02640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:25.106 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.126+0000 7ff6e5e02640 10 monclient(hunting): authenticate will time out at 1066.377707s 2024-03-29T11:21:25.109 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.130+0000 7ff6dffff640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7ff6e014ff40 0x7ff6e013b620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59493/0 (socket says 172.21.15.104:59493) 2024-03-29T11:21:25.109 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.130+0000 7ff6deffd640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7ff6e0149dc0 0x7ff6e013af10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:58445/0 (socket says 172.21.15.104:58445) 2024-03-29T11:21:25.109 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.130+0000 7ff6deffd640 1 -- 172.21.15.104:0/2465036688 learned_addr learned my addr 172.21.15.104:0/2465036688 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:21:25.109 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.131+0000 7ff6c7fff640 10 client.?.objecter ms_handle_connect 0x7ff6e014ff40 2024-03-29T11:21:25.109 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.131+0000 7ff6c7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:25.109 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.131+0000 7ff6c7fff640 10 client.?.objecter ms_handle_connect 0x7ff6e0149dc0 2024-03-29T11:21:25.109 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.131+0000 7ff6c7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:25.109 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.131+0000 7ff6c7fff640 1 -- 172.21.15.104:0/2465036688 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3848667402 0 0) 0x7ff6e0144a30 con 0x7ff6e014ff40 2024-03-29T11:21:25.109 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.131+0000 7ff6c7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:25.109 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.131+0000 7ff6c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:25.109 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.131+0000 7ff6c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:25.109 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.131+0000 7ff6c7fff640 10 monclient(hunting): my global_id is 4311 2024-03-29T11:21:25.110 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.131+0000 7ff6c7fff640 1 -- 172.21.15.104:0/2465036688 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6cc004eb0 con 0x7ff6e014ff40 2024-03-29T11:21:25.110 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.131+0000 7ff6c7fff640 1 -- 172.21.15.104:0/2465036688 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 340778954 0 0) 0x7ff6e00ac7f0 con 0x7ff6e0149dc0 2024-03-29T11:21:25.110 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.131+0000 7ff6c7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:25.110 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.131+0000 7ff6c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:25.110 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.131+0000 7ff6c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:25.110 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.131+0000 7ff6c7fff640 10 monclient(hunting): my global_id is 4316 2024-03-29T11:21:25.110 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.132+0000 7ff6c7fff640 1 -- 172.21.15.104:0/2465036688 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6e0144a30 con 0x7ff6e0149dc0 2024-03-29T11:21:25.110 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.132+0000 7ff6c7fff640 1 -- 172.21.15.104:0/2465036688 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 483346852 0 0) 0x7ff6cc004eb0 con 0x7ff6e014ff40 2024-03-29T11:21:25.110 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.132+0000 7ff6c7fff640 1 -- 172.21.15.104:0/2465036688 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6e00ac7f0 con 0x7ff6e014ff40 2024-03-29T11:21:25.110 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.132+0000 7ff6c7fff640 1 -- 172.21.15.104:0/2465036688 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7ff6d8001290 con 0x7ff6e014ff40 2024-03-29T11:21:25.110 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.132+0000 7ff6c7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:25.110 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.132+0000 7ff6c7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:25.110 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.132+0000 7ff6c7fff640 10 monclient(hunting): dump: 2024-03-29T11:21:25.110 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:21:25.110 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:21:25.111 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:25.111 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:25.111 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:25.111 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:21:25.111 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:21:25.111 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:21:25.111 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:21:25.111 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:21:25.111 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.132+0000 7ff6c7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:25.111 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.132+0000 7ff6c7fff640 1 -- 172.21.15.104:0/2465036688 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4031320349 0 0) 0x7ff6e0144a30 con 0x7ff6e0149dc0 2024-03-29T11:21:25.111 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.132+0000 7ff6c7fff640 1 -- 172.21.15.104:0/2465036688 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6cc004eb0 con 0x7ff6e0149dc0 2024-03-29T11:21:25.111 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.132+0000 7ff6c7fff640 1 -- 172.21.15.104:0/2465036688 <== mon.0 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3893374479 0 0) 0x7ff6e00ac7f0 con 0x7ff6e014ff40 2024-03-29T11:21:25.111 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.132+0000 7ff6c7fff640 1 -- 172.21.15.104:0/2465036688 >> v1:172.21.15.104:6790/0 conn(0x7ff6e0149dc0 legacy=0x7ff6e013af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:25.111 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.132+0000 7ff6c7fff640 1 -- 172.21.15.104:0/2465036688 >> v1:172.21.15.119:6789/0 conn(0x7ff6e00a9930 legacy=0x7ff6e01db640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:25.111 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.132+0000 7ff6c7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:21:25.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.132+0000 7ff6c7fff640 1 monclient: found mon.a 2024-03-29T11:21:25.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.132+0000 7ff6c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:25.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.132+0000 7ff6c7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:21:25.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.132+0000 7ff6c7fff640 1 -- 172.21.15.104:0/2465036688 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e0144510 con 0x7ff6e014ff40 2024-03-29T11:21:25.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.132+0000 7ff6c7fff640 10 monclient: _finish_auth 0 2024-03-29T11:21:25.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.132+0000 7ff6c7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:21:25.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.132+0000 7ff6c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:25.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.132+0000 7ff6e5e02640 5 monclient: authenticate success, global_id 4311 2024-03-29T11:21:25.114 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.132+0000 7ff6e5e02640 10 monclient: _renew_subs 2024-03-29T11:21:25.114 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.132+0000 7ff6e5e02640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:21:25.114 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.132+0000 7ff6e5e02640 1 -- 172.21.15.104:0/2465036688 --> v1:172.21.15.104:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6e0144760 con 0x7ff6e014ff40 2024-03-29T11:21:25.114 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.133+0000 7ff6e5e02640 10 client.4311.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:25.114 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.133+0000 7ff6e5e02640 10 monclient: _renew_subs 2024-03-29T11:21:25.114 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.133+0000 7ff6e5e02640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:21:25.114 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.133+0000 7ff6e5e02640 1 -- 172.21.15.104:0/2465036688 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6e00ad1b0 con 0x7ff6e014ff40 2024-03-29T11:21:25.114 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.133+0000 7ff6e5e02640 1 librados: init done 2024-03-29T11:21:25.114 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.133+0000 7ff6c7fff640 1 -- 172.21.15.104:0/2465036688 <== mon.0 v1:172.21.15.104:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6d8001d30 con 0x7ff6e014ff40 2024-03-29T11:21:25.114 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.133+0000 7ff6c7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:25.114 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.133+0000 7ff6c7fff640 1 -- 172.21.15.104:0/2465036688 <== mon.0 v1:172.21.15.104:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7ff6d8004fb0 con 0x7ff6e014ff40 2024-03-29T11:21:25.114 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.133+0000 7ff6c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:25.114 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.133+0000 7ff6c7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:25.114 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.133+0000 7ff6c7fff640 10 monclient: dump: 2024-03-29T11:21:25.115 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:21:25.115 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:21:25.115 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:25.115 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:25.115 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:25.115 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:21:25.115 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:21:25.115 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:21:25.115 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:21:25.115 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:21:25.115 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.135+0000 7ff6dd7fa640 4 set_mon_vals no callback set 2024-03-29T11:21:25.115 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.135+0000 7ff6c7fff640 1 -- 172.21.15.104:0/2465036688 <== mon.0 v1:172.21.15.104:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7ff6d801af80 con 0x7ff6e014ff40 2024-03-29T11:21:25.115 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.136+0000 7ff6c7fff640 1 -- 172.21.15.104:0/2465036688 <== mon.0 v1:172.21.15.104:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7ff6d8084c20 con 0x7ff6e014ff40 2024-03-29T11:21:25.115 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.136+0000 7ff6df7fe640 10 monclient: get_auth_request con 0x7ff6cc06df70 auth_method 0 2024-03-29T11:21:25.115 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.136+0000 7ff6c7fff640 10 client.4311.objecter ms_dispatch 0x7ff6e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:25.116 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.136+0000 7ff6c7fff640 3 client.4311.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:25.116 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.136+0000 7ff6c7fff640 3 client.4311.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:25.116 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.137+0000 7ff6c7fff640 20 client.4311.objecter dump_active .. 0 homeless 2024-03-29T11:21:25.116 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.137+0000 7ff6c7fff640 10 client.4311.objecter ms_handle_connect 0x7ff6cc06df70 2024-03-29T11:21:25.116 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.137+0000 7ff6e5e02640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:25.116 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.137+0000 7ff6e5e02640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:25.116 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.137+0000 7ff6e5e02640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:21:25.116 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.137+0000 7ff6e5e02640 1 -- 172.21.15.104:0/2465036688 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6c0005080 con 0x7ff6e014ff40 2024-03-29T11:21:25.121 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.145+0000 7ff6c7fff640 1 -- 172.21.15.104:0/2465036688 <== mon.0 v1:172.21.15.104:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff6d8050500 con 0x7ff6e014ff40 2024-03-29T11:21:25.121 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.145+0000 7ff6c7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:25.121 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.145+0000 7ff6c7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:25.450 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.474+0000 7ff6e5e02640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:25.451 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.474+0000 7ff6e5e02640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:25.451 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.474+0000 7ff6e5e02640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:21:25.451 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.474+0000 7ff6e5e02640 1 -- 172.21.15.104:0/2465036688 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6c0005330 con 0x7ff6e014ff40 2024-03-29T11:21:25.452 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.476+0000 7ff6c7fff640 1 -- 172.21.15.104:0/2465036688 <== mon.0 v1:172.21.15.104:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7ff6d8052430 con 0x7ff6e014ff40 2024-03-29T11:21:25.452 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.476+0000 7ff6c7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:25.452 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.476+0000 7ff6c7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:25.456 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:21:25.456 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:25.459 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.483+0000 7ff6e5e02640 10 librados: watch_flush enter 2024-03-29T11:21:25.459 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.483+0000 7ff6e5e02640 10 librados: watch_flush exit 2024-03-29T11:21:25.459 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.483+0000 7ff6e5e02640 20 client.4311.objecter shutdown clearing up homeless session... 2024-03-29T11:21:25.459 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.483+0000 7ff6e5e02640 10 client.4311.objecter successfully canceled tick 2024-03-29T11:21:25.460 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.483+0000 7ff6e5e02640 1 -- 172.21.15.104:0/2465036688 >> v1:172.21.15.104:6816/3341126660 conn(0x7ff6cc06df70 legacy=0x7ff6cc070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:25.460 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.483+0000 7ff6e5e02640 10 monclient: shutdown 2024-03-29T11:21:25.460 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.483+0000 7ff6e5e02640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:25.460 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.483+0000 7ff6e5e02640 1 -- 172.21.15.104:0/2465036688 >> v1:172.21.15.104:6789/0 conn(0x7ff6e014ff40 legacy=0x7ff6e013b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:25.460 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.484+0000 7ff6e5e02640 1 -- 172.21.15.104:0/2465036688 shutdown_connections 2024-03-29T11:21:25.460 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.484+0000 7ff6e5e02640 1 -- 172.21.15.104:0/2465036688 >> 172.21.15.104:0/2465036688 conn(0x7ff6e00b35e0 msgr2=0x7ff6e00b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:25.461 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.484+0000 7ff6e5e02640 1 -- 172.21.15.104:0/2465036688 shutdown_connections 2024-03-29T11:21:25.461 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.484+0000 7ff6e5e02640 1 -- 172.21.15.104:0/2465036688 wait complete. 2024-03-29T11:21:25.461 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:25.485+0000 7ff6e5e02640 1 librados: shutdown 2024-03-29T11:21:25.479 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:32.286 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:32.439 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.462+0000 7fc84b59e640 10 monclient: get_monmap_and_config 2024-03-29T11:21:32.439 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.462+0000 7fc84b59e640 10 monclient: build_initial_monmap 2024-03-29T11:21:32.439 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.462+0000 7fc84b59e640 10 monclient: monmap: 2024-03-29T11:21:32.439 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:21:32.439 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:21:32.439 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:21:32.463869+0000 2024-03-29T11:21:32.439 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:21:32.463869+0000 2024-03-29T11:21:32.440 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:32.440 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:21:32.440 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:21:32.440 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:21:32.440 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:21:32.440 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:21:32.441 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.465+0000 7fc84b59e640 1 Processor -- start 2024-03-29T11:21:32.442 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.465+0000 7fc84b59e640 1 -- start start 2024-03-29T11:21:32.442 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.465+0000 7fc84b59e640 10 monclient: init 2024-03-29T11:21:32.442 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.466+0000 7fc84b59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:32.444 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.466+0000 7fc84b59e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:21:32.444 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.466+0000 7fc84b59e640 10 monclient(hunting): picked mon.noname-c con 0x7fc84c0b2800 addr v1:172.21.15.119:6789/0 2024-03-29T11:21:32.444 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.466+0000 7fc84b59e640 10 monclient(hunting): picked mon.noname-b con 0x7fc84c0a9c00 addr v1:172.21.15.104:6790/0 2024-03-29T11:21:32.444 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.466+0000 7fc84b59e640 10 monclient(hunting): picked mon.noname-a con 0x7fc84c0b38c0 addr v1:172.21.15.104:6789/0 2024-03-29T11:21:32.444 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.466+0000 7fc84b59e640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc84c141eb0 con 0x7fc84c0b38c0 2024-03-29T11:21:32.444 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.466+0000 7fc84b59e640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc84c0b22d0 con 0x7fc84c0b2800 2024-03-29T11:21:32.445 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.466+0000 7fc84b59e640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc84c142d30 con 0x7fc84c0a9c00 2024-03-29T11:21:32.445 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.466+0000 7fc84b59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:32.445 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.466+0000 7fc84b59e640 10 monclient(hunting): authenticate will time out at 1073.717821s 2024-03-29T11:21:32.450 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.473+0000 7fc84ad9d640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fc84c0b38c0 0x7fc84c14d7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:48863/0 (socket says 172.21.15.104:48863) 2024-03-29T11:21:32.450 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.473+0000 7fc84ad9d640 1 -- 172.21.15.104:0/1686150697 learned_addr learned my addr 172.21.15.104:0/1686150697 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:21:32.451 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.474+0000 7fc84959a640 1 -- 172.21.15.104:0/1686150697 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1387445184 0 0) 0x7fc84c141eb0 con 0x7fc84c0b38c0 2024-03-29T11:21:32.451 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.474+0000 7fc84959a640 10 monclient(hunting): _negotiate 2024-03-29T11:21:32.451 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.474+0000 7fc84959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:32.451 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.474+0000 7fc84959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:32.451 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.474+0000 7fc84959a640 10 monclient(hunting): my global_id is 4314 2024-03-29T11:21:32.451 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.474+0000 7fc84959a640 1 -- 172.21.15.104:0/1686150697 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc834003520 con 0x7fc84c0b38c0 2024-03-29T11:21:32.453 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.476+0000 7fc84959a640 1 -- 172.21.15.104:0/1686150697 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3997166403 0 0) 0x7fc84c0b22d0 con 0x7fc84c0b2800 2024-03-29T11:21:32.453 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.476+0000 7fc84959a640 10 monclient(hunting): _negotiate 2024-03-29T11:21:32.453 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.476+0000 7fc84959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:32.453 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.476+0000 7fc84959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:32.453 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.476+0000 7fc84959a640 10 monclient(hunting): my global_id is 4291 2024-03-29T11:21:32.453 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.476+0000 7fc84959a640 1 -- 172.21.15.104:0/1686150697 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc84c141eb0 con 0x7fc84c0b2800 2024-03-29T11:21:32.453 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.476+0000 7fc84959a640 1 -- 172.21.15.104:0/1686150697 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4118038800 0 0) 0x7fc84c142d30 con 0x7fc84c0a9c00 2024-03-29T11:21:32.453 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.476+0000 7fc84959a640 10 monclient(hunting): _negotiate 2024-03-29T11:21:32.453 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.476+0000 7fc84959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:32.453 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.476+0000 7fc84959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:32.453 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.476+0000 7fc84959a640 10 monclient(hunting): my global_id is 4319 2024-03-29T11:21:32.453 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.476+0000 7fc84959a640 1 -- 172.21.15.104:0/1686150697 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc84c0b22d0 con 0x7fc84c0a9c00 2024-03-29T11:21:32.454 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.477+0000 7fc84959a640 1 -- 172.21.15.104:0/1686150697 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1252171692 0 0) 0x7fc84c0b22d0 con 0x7fc84c0a9c00 2024-03-29T11:21:32.454 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.477+0000 7fc84959a640 1 -- 172.21.15.104:0/1686150697 >> v1:172.21.15.119:6789/0 conn(0x7fc84c0b2800 legacy=0x7fc84c142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:32.454 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.477+0000 7fc84959a640 1 -- 172.21.15.104:0/1686150697 >> v1:172.21.15.104:6789/0 conn(0x7fc84c0b38c0 legacy=0x7fc84c14d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:32.454 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.477+0000 7fc84959a640 10 monclient: _finish_hunting 0 2024-03-29T11:21:32.454 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.477+0000 7fc84959a640 1 monclient: found mon.noname-b 2024-03-29T11:21:32.454 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.477+0000 7fc84959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:32.454 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.477+0000 7fc84959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.104:6790/0 2024-03-29T11:21:32.454 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.477+0000 7fc84959a640 1 -- 172.21.15.104:0/1686150697 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc84c0aaf90 con 0x7fc84c0a9c00 2024-03-29T11:21:32.454 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.477+0000 7fc84959a640 10 monclient: _finish_auth 0 2024-03-29T11:21:32.455 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.477+0000 7fc84959a640 10 monclient: _check_auth_tickets 2024-03-29T11:21:32.455 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.477+0000 7fc84959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:32.455 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.477+0000 7fc84959a640 1 -- 172.21.15.104:0/1686150697 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fc8400015b0 con 0x7fc84c0a9c00 2024-03-29T11:21:32.455 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.477+0000 7fc84959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:32.455 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.477+0000 7fc84959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:32.455 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.477+0000 7fc84959a640 10 monclient: dump: 2024-03-29T11:21:32.455 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:21:32.455 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:21:32.455 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:32.455 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:32.455 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:32.455 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:21:32.456 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:21:32.456 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:21:32.456 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:21:32.456 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:21:32.456 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.477+0000 7fc84b59e640 5 monclient: authenticate success, global_id 4319 2024-03-29T11:21:32.456 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.477+0000 7fc84b59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:32.457 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.479+0000 7fc84959a640 1 -- 172.21.15.104:0/1686150697 <== mon.2 v1:172.21.15.104:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc840001710 con 0x7fc84c0a9c00 2024-03-29T11:21:32.457 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.479+0000 7fc84959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:32.457 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.479+0000 7fc84b59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:32.457 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.479+0000 7fc84b59e640 4 set_mon_vals no callback set 2024-03-29T11:21:32.457 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.479+0000 7fc84959a640 1 -- 172.21.15.104:0/1686150697 <== mon.2 v1:172.21.15.104:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fc840001b20 con 0x7fc84c0a9c00 2024-03-29T11:21:32.457 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.479+0000 7fc84959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:32.457 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.480+0000 7fc84959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:32.457 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.480+0000 7fc84959a640 10 monclient: dump: 2024-03-29T11:21:32.457 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:21:32.457 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:21:32.457 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:32.457 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:32.458 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:32.458 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:21:32.458 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:21:32.458 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:21:32.458 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:21:32.458 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:21:32.458 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.480+0000 7fc84b59e640 10 monclient: shutdown 2024-03-29T11:21:32.458 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.480+0000 7fc84b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:32.458 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.480+0000 7fc84b59e640 1 -- 172.21.15.104:0/1686150697 >> v1:172.21.15.104:6790/0 conn(0x7fc84c0a9c00 legacy=0x7fc84c13bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:32.463 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.481+0000 7fc84b59e640 1 -- 172.21.15.104:0/1686150697 shutdown_connections 2024-03-29T11:21:32.463 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.481+0000 7fc84b59e640 1 -- 172.21.15.104:0/1686150697 >> 172.21.15.104:0/1686150697 conn(0x7fc84c066a00 msgr2=0x7fc84c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:32.463 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.481+0000 7fc84b59e640 1 -- 172.21.15.104:0/1686150697 shutdown_connections 2024-03-29T11:21:32.463 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.481+0000 7fc84b59e640 1 -- 172.21.15.104:0/1686150697 wait complete. 2024-03-29T11:21:32.463 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.482+0000 7fc84b59e640 10 monclient: build_initial_monmap 2024-03-29T11:21:32.463 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.482+0000 7fc84b59e640 10 monclient: monmap: 2024-03-29T11:21:32.463 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:21:32.463 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:32.463 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:21:32.463 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:21:32.463 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:32.463 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:21:32.463 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:21:32.464 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:21:32.464 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:21:32.464 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:21:32.464 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.483+0000 7fc84b59e640 1 librados: starting msgr at 2024-03-29T11:21:32.464 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.483+0000 7fc84b59e640 1 librados: starting objecter 2024-03-29T11:21:32.464 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.483+0000 7fc84b59e640 1 Processor -- start 2024-03-29T11:21:32.464 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.483+0000 7fc84b59e640 1 -- start start 2024-03-29T11:21:32.464 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.483+0000 7fc84b59e640 1 librados: setting wanted keys 2024-03-29T11:21:32.464 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.483+0000 7fc84b59e640 1 librados: calling monclient init 2024-03-29T11:21:32.464 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.483+0000 7fc84b59e640 10 monclient: init 2024-03-29T11:21:32.464 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.484+0000 7fc84b59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:32.464 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.484+0000 7fc84b59e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:32.464 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.484+0000 7fc84b59e640 10 monclient(hunting): picked mon.noname-c con 0x7fc84c0a9c00 addr v1:172.21.15.104:6790/0 2024-03-29T11:21:32.464 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.484+0000 7fc84b59e640 10 monclient(hunting): picked mon.noname-b con 0x7fc84c0b2800 addr v1:172.21.15.119:6789/0 2024-03-29T11:21:32.464 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.484+0000 7fc84b59e640 10 monclient(hunting): picked mon.noname-a con 0x7fc84c0b38c0 addr v1:172.21.15.104:6789/0 2024-03-29T11:21:32.465 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.484+0000 7fc84b59e640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc84c12b710 con 0x7fc84c0b38c0 2024-03-29T11:21:32.465 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.484+0000 7fc84b59e640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc84c070f20 con 0x7fc84c0b2800 2024-03-29T11:21:32.465 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.484+0000 7fc84b59e640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc84c071190 con 0x7fc84c0a9c00 2024-03-29T11:21:32.465 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.484+0000 7fc84b59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:32.465 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.484+0000 7fc84b59e640 10 monclient(hunting): authenticate will time out at 1073.735165s 2024-03-29T11:21:32.467 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.489+0000 7fc84ad9d640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fc84c0b38c0 0x7fc84c085060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:49173/0 (socket says 172.21.15.104:49173) 2024-03-29T11:21:32.467 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.489+0000 7fc84ad9d640 1 -- 172.21.15.104:0/3231297573 learned_addr learned my addr 172.21.15.104:0/3231297573 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:21:32.467 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.489+0000 7fc83affd640 10 client.?.objecter ms_handle_connect 0x7fc84c0b38c0 2024-03-29T11:21:32.467 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.489+0000 7fc83affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:32.467 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.489+0000 7fc83affd640 1 -- 172.21.15.104:0/3231297573 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 500296926 0 0) 0x7fc84c12b710 con 0x7fc84c0b38c0 2024-03-29T11:21:32.467 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.489+0000 7fc83affd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:32.467 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.489+0000 7fc83affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:32.467 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.489+0000 7fc83affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:32.467 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.489+0000 7fc83affd640 10 monclient(hunting): my global_id is 4317 2024-03-29T11:21:32.467 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.489+0000 7fc83affd640 1 -- 172.21.15.104:0/3231297573 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc82c004eb0 con 0x7fc84c0b38c0 2024-03-29T11:21:32.468 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.490+0000 7fc83affd640 1 -- 172.21.15.104:0/3231297573 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 832812038 0 0) 0x7fc82c004eb0 con 0x7fc84c0b38c0 2024-03-29T11:21:32.468 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.490+0000 7fc83affd640 1 -- 172.21.15.104:0/3231297573 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc84c12b710 con 0x7fc84c0b38c0 2024-03-29T11:21:32.468 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.490+0000 7fc83affd640 1 -- 172.21.15.104:0/3231297573 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fc83c0017f0 con 0x7fc84c0b38c0 2024-03-29T11:21:32.468 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.490+0000 7fc83affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:32.468 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.490+0000 7fc83affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:32.468 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.490+0000 7fc83affd640 10 monclient(hunting): dump: 2024-03-29T11:21:32.468 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:21:32.468 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:21:32.468 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:32.468 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:32.468 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:32.468 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:21:32.468 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:21:32.469 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:21:32.469 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:21:32.469 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:21:32.469 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.490+0000 7fc83affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:32.473 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.491+0000 7fc83affd640 10 client.?.objecter ms_handle_connect 0x7fc84c0a9c00 2024-03-29T11:21:32.473 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.491+0000 7fc83affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:32.473 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.491+0000 7fc83affd640 1 -- 172.21.15.104:0/3231297573 <== mon.0 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 818613770 0 0) 0x7fc84c12b710 con 0x7fc84c0b38c0 2024-03-29T11:21:32.473 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.491+0000 7fc83affd640 1 -- 172.21.15.104:0/3231297573 >> v1:172.21.15.104:6790/0 conn(0x7fc84c0a9c00 legacy=0x7fc84c13f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:32.473 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.491+0000 7fc83affd640 1 -- 172.21.15.104:0/3231297573 >> v1:172.21.15.119:6789/0 conn(0x7fc84c0b2800 legacy=0x7fc84c06f6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:21:32.473 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.491+0000 7fc83affd640 10 monclient: _finish_hunting 0 2024-03-29T11:21:32.473 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.491+0000 7fc83affd640 1 monclient: found mon.a 2024-03-29T11:21:32.473 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.491+0000 7fc83affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:32.474 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.491+0000 7fc83affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:21:32.474 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.491+0000 7fc83affd640 1 -- 172.21.15.104:0/3231297573 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc84c071470 con 0x7fc84c0b38c0 2024-03-29T11:21:32.474 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.491+0000 7fc83affd640 10 monclient: _finish_auth 0 2024-03-29T11:21:32.474 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.491+0000 7fc83affd640 10 monclient: _check_auth_tickets 2024-03-29T11:21:32.474 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.491+0000 7fc83affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:32.474 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.492+0000 7fc83affd640 1 -- 172.21.15.104:0/3231297573 <== mon.0 v1:172.21.15.104:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc83c001d60 con 0x7fc84c0b38c0 2024-03-29T11:21:32.474 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.492+0000 7fc83affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:32.474 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.492+0000 7fc83affd640 1 -- 172.21.15.104:0/3231297573 <== mon.0 v1:172.21.15.104:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fc83c004980 con 0x7fc84c0b38c0 2024-03-29T11:21:32.474 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.492+0000 7fc83affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:32.474 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.492+0000 7fc83affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:32.474 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.492+0000 7fc83affd640 10 monclient: dump: 2024-03-29T11:21:32.474 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:21:32.474 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:21:32.475 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:32.475 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:32.475 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:32.475 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:21:32.475 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:21:32.475 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:21:32.475 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:21:32.475 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:21:32.475 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.492+0000 7fc83bfff640 4 set_mon_vals no callback set 2024-03-29T11:21:32.475 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.492+0000 7fc84b59e640 5 monclient: authenticate success, global_id 4317 2024-03-29T11:21:32.475 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.492+0000 7fc84b59e640 10 monclient: _renew_subs 2024-03-29T11:21:32.475 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.492+0000 7fc84b59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:21:32.475 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.492+0000 7fc84b59e640 1 -- 172.21.15.104:0/3231297573 --> v1:172.21.15.104:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc84c071c30 con 0x7fc84c0b38c0 2024-03-29T11:21:32.475 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.492+0000 7fc84b59e640 10 client.4317.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:32.475 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.492+0000 7fc84b59e640 10 monclient: _renew_subs 2024-03-29T11:21:32.476 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.492+0000 7fc84b59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:21:32.476 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.492+0000 7fc84b59e640 1 -- 172.21.15.104:0/3231297573 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc84c12e590 con 0x7fc84c0b38c0 2024-03-29T11:21:32.476 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.492+0000 7fc84b59e640 1 librados: init done 2024-03-29T11:21:32.476 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.494+0000 7fc84b59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:32.476 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.494+0000 7fc84b59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:32.476 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.494+0000 7fc84b59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:21:32.476 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.494+0000 7fc84b59e640 1 -- 172.21.15.104:0/3231297573 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc84c0b5ab0 con 0x7fc84c0b38c0 2024-03-29T11:21:32.476 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.495+0000 7fc83affd640 1 -- 172.21.15.104:0/3231297573 <== mon.0 v1:172.21.15.104:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7fc83c004ae0 con 0x7fc84c0b38c0 2024-03-29T11:21:32.476 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.495+0000 7fc83affd640 1 -- 172.21.15.104:0/3231297573 <== mon.0 v1:172.21.15.104:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7fc83c084a60 con 0x7fc84c0b38c0 2024-03-29T11:21:32.476 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.495+0000 7fc83affd640 10 client.4317.objecter ms_dispatch 0x7fc84c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:32.476 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.495+0000 7fc83affd640 3 client.4317.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:32.476 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.495+0000 7fc83affd640 3 client.4317.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:32.476 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.495+0000 7fc83affd640 20 client.4317.objecter dump_active .. 0 homeless 2024-03-29T11:21:32.476 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.496+0000 7fc84a59c640 10 monclient: get_auth_request con 0x7fc82c06dc20 auth_method 0 2024-03-29T11:21:32.477 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.496+0000 7fc83affd640 10 client.4317.objecter ms_handle_connect 0x7fc82c06dc20 2024-03-29T11:21:32.483 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.505+0000 7fc83affd640 1 -- 172.21.15.104:0/3231297573 <== mon.0 v1:172.21.15.104:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc83c050170 con 0x7fc84c0b38c0 2024-03-29T11:21:32.483 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.505+0000 7fc83affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:32.483 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.505+0000 7fc83affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:32.793 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.816+0000 7fc84b59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:32.793 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.816+0000 7fc84b59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:32.793 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.817+0000 7fc84b59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:21:32.793 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.817+0000 7fc84b59e640 1 -- 172.21.15.104:0/3231297573 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc84c132d20 con 0x7fc84c0b38c0 2024-03-29T11:21:32.796 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.820+0000 7fc83affd640 1 -- 172.21.15.104:0/3231297573 <== mon.0 v1:172.21.15.104:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7fc83c052270 con 0x7fc84c0b38c0 2024-03-29T11:21:32.796 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.820+0000 7fc83affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:32.796 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.820+0000 7fc83affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:32.797 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:21:32.797 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:32.801 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.824+0000 7fc84b59e640 10 librados: watch_flush enter 2024-03-29T11:21:32.801 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.824+0000 7fc84b59e640 10 librados: watch_flush exit 2024-03-29T11:21:32.801 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.825+0000 7fc84b59e640 20 client.4317.objecter shutdown clearing up homeless session... 2024-03-29T11:21:32.801 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.825+0000 7fc84b59e640 10 client.4317.objecter successfully canceled tick 2024-03-29T11:21:32.803 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.826+0000 7fc84b59e640 1 -- 172.21.15.104:0/3231297573 >> v1:172.21.15.104:6816/3341126660 conn(0x7fc82c06dc20 legacy=0x7fc82c0700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:32.803 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.826+0000 7fc84b59e640 10 monclient: shutdown 2024-03-29T11:21:32.803 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.826+0000 7fc84b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:32.803 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.826+0000 7fc84b59e640 1 -- 172.21.15.104:0/3231297573 >> v1:172.21.15.104:6789/0 conn(0x7fc84c0b38c0 legacy=0x7fc84c085060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:32.803 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.827+0000 7fc84b59e640 1 -- 172.21.15.104:0/3231297573 shutdown_connections 2024-03-29T11:21:32.803 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.827+0000 7fc84b59e640 1 -- 172.21.15.104:0/3231297573 >> 172.21.15.104:0/3231297573 conn(0x7fc84c066a00 msgr2=0x7fc84c131e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:32.804 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.828+0000 7fc84b59e640 1 -- 172.21.15.104:0/3231297573 shutdown_connections 2024-03-29T11:21:32.805 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.829+0000 7fc84b59e640 1 -- 172.21.15.104:0/3231297573 wait complete. 2024-03-29T11:21:32.806 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:32.830+0000 7fc84b59e640 1 librados: shutdown 2024-03-29T11:21:32.825 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:39.632 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:39.772 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.796+0000 7f129120f640 10 monclient: get_monmap_and_config 2024-03-29T11:21:39.772 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.796+0000 7f129120f640 10 monclient: build_initial_monmap 2024-03-29T11:21:39.772 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.796+0000 7f129120f640 10 monclient: monmap: 2024-03-29T11:21:39.773 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:21:39.773 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:21:39.773 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:21:39.797136+0000 2024-03-29T11:21:39.773 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:21:39.797136+0000 2024-03-29T11:21:39.773 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:39.773 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:21:39.773 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:21:39.773 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:21:39.773 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:21:39.773 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:21:39.774 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.797+0000 7f129120f640 1 Processor -- start 2024-03-29T11:21:39.774 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.797+0000 7f129120f640 1 -- start start 2024-03-29T11:21:39.774 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.797+0000 7f129120f640 10 monclient: init 2024-03-29T11:21:39.776 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.798+0000 7f129120f640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:39.776 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.798+0000 7f129120f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:39.776 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.798+0000 7f129120f640 10 monclient(hunting): picked mon.noname-b con 0x7f128c0b0860 addr v1:172.21.15.104:6790/0 2024-03-29T11:21:39.776 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.799+0000 7f129120f640 10 monclient(hunting): picked mon.noname-c con 0x7f128c13ef30 addr v1:172.21.15.119:6789/0 2024-03-29T11:21:39.776 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.799+0000 7f129120f640 10 monclient(hunting): picked mon.noname-a con 0x7f128c0af8e0 addr v1:172.21.15.104:6789/0 2024-03-29T11:21:39.776 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.799+0000 7f128ad9d640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f128c0b0860 0x7f128c13e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:59188/0 (socket says 172.21.15.104:59188) 2024-03-29T11:21:39.776 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.799+0000 7f128ad9d640 1 -- 172.21.15.104:0/2870087088 learned_addr learned my addr 172.21.15.104:0/2870087088 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:21:39.776 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.799+0000 7f129120f640 1 -- 172.21.15.104:0/2870087088 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f128c0b0c40 con 0x7f128c0af8e0 2024-03-29T11:21:39.777 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.799+0000 7f129120f640 1 -- 172.21.15.104:0/2870087088 --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f128c0aed10 con 0x7f128c13ef30 2024-03-29T11:21:39.777 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.799+0000 7f129120f640 1 -- 172.21.15.104:0/2870087088 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f128c0afd50 con 0x7f128c0b0860 2024-03-29T11:21:39.777 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.799+0000 7f129120f640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:39.777 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.799+0000 7f129120f640 10 monclient(hunting): authenticate will time out at 1081.050355s 2024-03-29T11:21:39.777 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.799+0000 7f1289d9b640 1 -- 172.21.15.104:0/2870087088 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2325739364 0 0) 0x7f128c0afd50 con 0x7f128c0b0860 2024-03-29T11:21:39.777 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.799+0000 7f1289d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:21:39.777 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.799+0000 7f1289d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:39.777 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.799+0000 7f1289d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:39.777 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.799+0000 7f1289d9b640 10 monclient(hunting): my global_id is 4325 2024-03-29T11:21:39.777 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.799+0000 7f1289d9b640 1 -- 172.21.15.104:0/2870087088 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1278003520 con 0x7f128c0b0860 2024-03-29T11:21:39.777 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.800+0000 7f1289d9b640 1 -- 172.21.15.104:0/2870087088 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3970241969 0 0) 0x7f1278003520 con 0x7f128c0b0860 2024-03-29T11:21:39.777 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.800+0000 7f1289d9b640 1 -- 172.21.15.104:0/2870087088 >> v1:172.21.15.119:6789/0 conn(0x7f128c13ef30 legacy=0x7f128c0af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:39.777 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.800+0000 7f1289d9b640 1 -- 172.21.15.104:0/2870087088 >> v1:172.21.15.104:6789/0 conn(0x7f128c0af8e0 legacy=0x7f128c149a70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:21:39.779 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.802+0000 7f1289d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:21:39.779 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.802+0000 7f1289d9b640 1 monclient: found mon.noname-b 2024-03-29T11:21:39.779 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.802+0000 7f1289d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:39.779 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.802+0000 7f1289d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.104:6790/0 2024-03-29T11:21:39.779 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.802+0000 7f1289d9b640 1 -- 172.21.15.104:0/2870087088 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f128c0ab1b0 con 0x7f128c0b0860 2024-03-29T11:21:39.780 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.802+0000 7f1289d9b640 10 monclient: _finish_auth 0 2024-03-29T11:21:39.780 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.802+0000 7f1289d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:21:39.780 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.802+0000 7f1289d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:39.780 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.802+0000 7f1289d9b640 1 -- 172.21.15.104:0/2870087088 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f127c001a10 con 0x7f128c0b0860 2024-03-29T11:21:39.780 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.802+0000 7f1289d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:39.780 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.802+0000 7f1289d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:39.780 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.802+0000 7f1289d9b640 10 monclient: dump: 2024-03-29T11:21:39.780 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:21:39.780 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:21:39.780 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:39.780 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:39.780 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:39.780 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:21:39.780 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:21:39.780 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:21:39.781 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:21:39.781 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:21:39.781 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.802+0000 7f129120f640 5 monclient: authenticate success, global_id 4325 2024-03-29T11:21:39.781 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.802+0000 7f129120f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:39.781 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.802+0000 7f1289d9b640 1 -- 172.21.15.104:0/2870087088 <== mon.2 v1:172.21.15.104:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f127c001be0 con 0x7f128c0b0860 2024-03-29T11:21:39.781 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.802+0000 7f1289d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:39.781 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.802+0000 7f1289d9b640 1 -- 172.21.15.104:0/2870087088 <== mon.2 v1:172.21.15.104:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f127c002030 con 0x7f128c0b0860 2024-03-29T11:21:39.781 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.803+0000 7f1289d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:39.781 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.803+0000 7f1289d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:39.781 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.803+0000 7f1289d9b640 10 monclient: dump: 2024-03-29T11:21:39.781 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:21:39.781 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:21:39.781 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:39.781 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:39.781 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:39.781 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:21:39.782 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:21:39.782 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:21:39.782 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:21:39.782 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:21:39.782 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.803+0000 7f129120f640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:39.782 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.803+0000 7f129120f640 4 set_mon_vals no callback set 2024-03-29T11:21:39.783 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.804+0000 7f129120f640 10 monclient: shutdown 2024-03-29T11:21:39.784 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.804+0000 7f129120f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:39.784 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.804+0000 7f129120f640 1 -- 172.21.15.104:0/2870087088 >> v1:172.21.15.104:6790/0 conn(0x7f128c0b0860 legacy=0x7f128c13e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:39.784 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.804+0000 7f129120f640 1 -- 172.21.15.104:0/2870087088 shutdown_connections 2024-03-29T11:21:39.784 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.804+0000 7f129120f640 1 -- 172.21.15.104:0/2870087088 >> 172.21.15.104:0/2870087088 conn(0x7f128c06b660 msgr2=0x7f128c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:39.784 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.804+0000 7f129120f640 1 -- 172.21.15.104:0/2870087088 shutdown_connections 2024-03-29T11:21:39.784 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.804+0000 7f129120f640 1 -- 172.21.15.104:0/2870087088 wait complete. 2024-03-29T11:21:39.784 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.806+0000 7f129120f640 10 monclient: build_initial_monmap 2024-03-29T11:21:39.784 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.806+0000 7f129120f640 10 monclient: monmap: 2024-03-29T11:21:39.784 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:21:39.784 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:39.784 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:21:39.784 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:21:39.784 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:39.784 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:21:39.785 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:21:39.785 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:21:39.785 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:21:39.785 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:21:39.785 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.806+0000 7f129120f640 1 librados: starting msgr at 2024-03-29T11:21:39.785 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.806+0000 7f129120f640 1 librados: starting objecter 2024-03-29T11:21:39.785 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.807+0000 7f129120f640 1 Processor -- start 2024-03-29T11:21:39.785 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.807+0000 7f129120f640 1 -- start start 2024-03-29T11:21:39.785 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.807+0000 7f129120f640 1 librados: setting wanted keys 2024-03-29T11:21:39.785 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.807+0000 7f129120f640 1 librados: calling monclient init 2024-03-29T11:21:39.785 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.807+0000 7f129120f640 10 monclient: init 2024-03-29T11:21:39.792 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.808+0000 7f129120f640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:39.792 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.808+0000 7f129120f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:21:39.792 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.808+0000 7f129120f640 10 monclient(hunting): picked mon.noname-a con 0x7f128c0af8e0 addr v1:172.21.15.104:6789/0 2024-03-29T11:21:39.792 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.808+0000 7f129120f640 10 monclient(hunting): picked mon.noname-b con 0x7f128c0b0860 addr v1:172.21.15.119:6789/0 2024-03-29T11:21:39.792 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.808+0000 7f129120f640 10 monclient(hunting): picked mon.noname-c con 0x7f128c13ef30 addr v1:172.21.15.104:6790/0 2024-03-29T11:21:39.792 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.808+0000 7f129120f640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f128c135ec0 con 0x7f128c0af8e0 2024-03-29T11:21:39.792 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.808+0000 7f129120f640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f128c137ec0 con 0x7f128c0b0860 2024-03-29T11:21:39.792 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.808+0000 7f129120f640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f128c1381a0 con 0x7f128c13ef30 2024-03-29T11:21:39.792 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.808+0000 7f129120f640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:39.792 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.808+0000 7f129120f640 10 monclient(hunting): authenticate will time out at 1081.059539s 2024-03-29T11:21:39.792 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.808+0000 7f128b59e640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f128c13ef30 0x7f128c0b2b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:38312/0 (socket says 172.21.15.104:38312) 2024-03-29T11:21:39.793 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.808+0000 7f128b59e640 1 -- 172.21.15.104:0/3316543992 learned_addr learned my addr 172.21.15.104:0/3316543992 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:21:39.793 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.808+0000 7f12837fe640 10 client.?.objecter ms_handle_connect 0x7f128c13ef30 2024-03-29T11:21:39.793 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.808+0000 7f12837fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:39.793 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.808+0000 7f12837fe640 10 client.?.objecter ms_handle_connect 0x7f128c0b0860 2024-03-29T11:21:39.793 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.808+0000 7f12837fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:39.793 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.808+0000 7f12837fe640 1 -- 172.21.15.104:0/3316543992 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 724706317 0 0) 0x7f128c1381a0 con 0x7f128c13ef30 2024-03-29T11:21:39.793 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.808+0000 7f12837fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:39.793 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.808+0000 7f12837fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:39.793 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.808+0000 7f12837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:39.793 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.808+0000 7f12837fe640 10 monclient(hunting): my global_id is 4328 2024-03-29T11:21:39.793 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.808+0000 7f12837fe640 1 -- 172.21.15.104:0/3316543992 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f126c004eb0 con 0x7f128c13ef30 2024-03-29T11:21:39.793 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.808+0000 7f12837fe640 1 -- 172.21.15.104:0/3316543992 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2525826618 0 0) 0x7f128c137ec0 con 0x7f128c0b0860 2024-03-29T11:21:39.793 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.809+0000 7f12837fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:39.793 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.809+0000 7f12837fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:39.793 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.809+0000 7f12837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:39.794 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.809+0000 7f12837fe640 10 monclient(hunting): my global_id is 4294 2024-03-29T11:21:39.794 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.809+0000 7f12837fe640 1 -- 172.21.15.104:0/3316543992 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f128c1381a0 con 0x7f128c0b0860 2024-03-29T11:21:39.794 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.809+0000 7f12837fe640 1 -- 172.21.15.104:0/3316543992 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 305181235 0 0) 0x7f126c004eb0 con 0x7f128c13ef30 2024-03-29T11:21:39.794 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.809+0000 7f12837fe640 1 -- 172.21.15.104:0/3316543992 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f128c137ec0 con 0x7f128c13ef30 2024-03-29T11:21:39.794 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.809+0000 7f12837fe640 1 -- 172.21.15.104:0/3316543992 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f12840018e0 con 0x7f128c13ef30 2024-03-29T11:21:39.794 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.809+0000 7f12837fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:39.794 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.809+0000 7f12837fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:39.794 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.809+0000 7f12837fe640 10 monclient(hunting): dump: 2024-03-29T11:21:39.794 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:21:39.794 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:21:39.794 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:39.794 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:39.794 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:39.794 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:21:39.794 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:21:39.795 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:21:39.795 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:21:39.795 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:21:39.795 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.809+0000 7f12837fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:39.795 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.809+0000 7f12837fe640 1 -- 172.21.15.104:0/3316543992 <== mon.2 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3088180057 0 0) 0x7f128c137ec0 con 0x7f128c13ef30 2024-03-29T11:21:39.795 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.809+0000 7f12837fe640 1 -- 172.21.15.104:0/3316543992 >> v1:172.21.15.119:6789/0 conn(0x7f128c0b0860 legacy=0x7f128c0b2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:39.795 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.809+0000 7f12837fe640 1 -- 172.21.15.104:0/3316543992 >> v1:172.21.15.104:6789/0 conn(0x7f128c0af8e0 legacy=0x7f128c136bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:21:39.795 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.810+0000 7f12837fe640 10 monclient: _finish_hunting 0 2024-03-29T11:21:39.795 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.810+0000 7f12837fe640 1 monclient: found mon.c 2024-03-29T11:21:39.795 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.810+0000 7f12837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:39.795 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.810+0000 7f12837fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:21:39.795 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.810+0000 7f12837fe640 1 -- 172.21.15.104:0/3316543992 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f128c138480 con 0x7f128c13ef30 2024-03-29T11:21:39.795 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.810+0000 7f12837fe640 10 monclient: _finish_auth 0 2024-03-29T11:21:39.795 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.810+0000 7f12837fe640 10 monclient: _check_auth_tickets 2024-03-29T11:21:39.795 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.810+0000 7f12837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:39.796 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.811+0000 7f129120f640 5 monclient: authenticate success, global_id 4328 2024-03-29T11:21:39.796 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.811+0000 7f129120f640 10 monclient: _renew_subs 2024-03-29T11:21:39.796 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.811+0000 7f129120f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:21:39.796 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.811+0000 7f129120f640 1 -- 172.21.15.104:0/3316543992 --> v1:172.21.15.104:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f128c1372d0 con 0x7f128c13ef30 2024-03-29T11:21:39.796 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.811+0000 7f129120f640 10 client.4328.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:39.796 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.811+0000 7f129120f640 10 monclient: _renew_subs 2024-03-29T11:21:39.796 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.811+0000 7f129120f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:21:39.796 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.811+0000 7f129120f640 1 -- 172.21.15.104:0/3316543992 --> v1:172.21.15.104:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f128c1e0cd0 con 0x7f128c13ef30 2024-03-29T11:21:39.796 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.811+0000 7f129120f640 1 librados: init done 2024-03-29T11:21:39.796 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.812+0000 7f12837fe640 1 -- 172.21.15.104:0/3316543992 <== mon.2 v1:172.21.15.104:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f12840035f0 con 0x7f128c13ef30 2024-03-29T11:21:39.796 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.812+0000 7f12837fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:39.796 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.812+0000 7f12837fe640 1 -- 172.21.15.104:0/3316543992 <== mon.2 v1:172.21.15.104:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f1284003940 con 0x7f128c13ef30 2024-03-29T11:21:39.796 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.812+0000 7f12837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:39.796 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.812+0000 7f129120f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:39.796 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.812+0000 7f12837fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:39.797 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.812+0000 7f12837fe640 10 monclient: dump: 2024-03-29T11:21:39.797 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:21:39.797 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:21:39.797 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:39.797 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:39.797 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:39.797 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:21:39.797 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:21:39.797 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:21:39.797 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:21:39.797 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:21:39.797 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.812+0000 7f128959a640 4 set_mon_vals no callback set 2024-03-29T11:21:39.797 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.813+0000 7f12837fe640 1 -- 172.21.15.104:0/3316543992 <== mon.2 v1:172.21.15.104:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f1284003b60 con 0x7f128c13ef30 2024-03-29T11:21:39.797 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.814+0000 7f12837fe640 1 -- 172.21.15.104:0/3316543992 <== mon.2 v1:172.21.15.104:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f1284084aa0 con 0x7f128c13ef30 2024-03-29T11:21:39.797 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.814+0000 7f12837fe640 10 client.4328.objecter ms_dispatch 0x7f128c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:39.797 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.814+0000 7f12837fe640 3 client.4328.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:39.798 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.814+0000 7f12837fe640 3 client.4328.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:39.798 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.814+0000 7f12837fe640 20 client.4328.objecter dump_active .. 0 homeless 2024-03-29T11:21:39.798 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.814+0000 7f129120f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:39.798 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.814+0000 7f129120f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:21:39.798 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.814+0000 7f129120f640 1 -- 172.21.15.104:0/3316543992 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f128c136090 con 0x7f128c13ef30 2024-03-29T11:21:39.798 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.814+0000 7f128ad9d640 10 monclient: get_auth_request con 0x7f126c06dd70 auth_method 0 2024-03-29T11:21:39.798 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.814+0000 7f12837fe640 10 client.4328.objecter ms_handle_connect 0x7f126c06dd70 2024-03-29T11:21:39.799 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.823+0000 7f12837fe640 1 -- 172.21.15.104:0/3316543992 <== mon.2 v1:172.21.15.104:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f128c136090 con 0x7f128c13ef30 2024-03-29T11:21:39.799 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.823+0000 7f12837fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:39.799 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:39.823+0000 7f12837fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:40.102 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:40.125+0000 7f129120f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:40.102 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:40.125+0000 7f129120f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:40.102 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:40.125+0000 7f129120f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:21:40.102 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:40.125+0000 7f129120f640 1 -- 172.21.15.104:0/3316543992 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f128c136090 con 0x7f128c13ef30 2024-03-29T11:21:40.102 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:40.126+0000 7f12837fe640 1 -- 172.21.15.104:0/3316543992 <== mon.2 v1:172.21.15.104:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f128c136090 con 0x7f128c13ef30 2024-03-29T11:21:40.102 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:40.126+0000 7f12837fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:40.103 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:40.126+0000 7f12837fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:40.103 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:21:40.103 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:40.105 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:40.129+0000 7f129120f640 10 librados: watch_flush enter 2024-03-29T11:21:40.105 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:40.129+0000 7f129120f640 10 librados: watch_flush exit 2024-03-29T11:21:40.105 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:40.129+0000 7f129120f640 20 client.4328.objecter shutdown clearing up homeless session... 2024-03-29T11:21:40.105 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:40.129+0000 7f129120f640 10 client.4328.objecter successfully canceled tick 2024-03-29T11:21:40.106 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:40.129+0000 7f129120f640 1 -- 172.21.15.104:0/3316543992 >> v1:172.21.15.104:6816/3341126660 conn(0x7f126c06dd70 legacy=0x7f126c070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:40.106 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:40.129+0000 7f129120f640 10 monclient: shutdown 2024-03-29T11:21:40.106 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:40.129+0000 7f129120f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:40.106 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:40.129+0000 7f129120f640 1 -- 172.21.15.104:0/3316543992 >> v1:172.21.15.104:6790/0 conn(0x7f128c13ef30 legacy=0x7f128c0b2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:40.106 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:40.129+0000 7f129120f640 1 -- 172.21.15.104:0/3316543992 shutdown_connections 2024-03-29T11:21:40.106 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:40.129+0000 7f129120f640 1 -- 172.21.15.104:0/3316543992 >> 172.21.15.104:0/3316543992 conn(0x7f128c06b660 msgr2=0x7f128c13e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:40.106 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:40.129+0000 7f129120f640 1 -- 172.21.15.104:0/3316543992 shutdown_connections 2024-03-29T11:21:40.106 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:40.129+0000 7f129120f640 1 -- 172.21.15.104:0/3316543992 wait complete. 2024-03-29T11:21:40.106 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:40.129+0000 7f129120f640 1 librados: shutdown 2024-03-29T11:21:40.120 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:46.927 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:47.077 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.099+0000 7fc1b4ce4640 10 monclient: get_monmap_and_config 2024-03-29T11:21:47.077 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.099+0000 7fc1b4ce4640 10 monclient: build_initial_monmap 2024-03-29T11:21:47.077 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.099+0000 7fc1b4ce4640 10 monclient: monmap: 2024-03-29T11:21:47.077 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:21:47.077 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:21:47.077 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:21:47.100783+0000 2024-03-29T11:21:47.077 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:21:47.100783+0000 2024-03-29T11:21:47.077 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:47.077 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:21:47.077 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:21:47.077 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:21:47.077 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:21:47.078 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:21:47.078 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.100+0000 7fc1b4ce4640 1 Processor -- start 2024-03-29T11:21:47.078 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.100+0000 7fc1b4ce4640 1 -- start start 2024-03-29T11:21:47.078 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.100+0000 7fc1b4ce4640 10 monclient: init 2024-03-29T11:21:47.081 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.101+0000 7fc1b4ce4640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:47.081 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.102+0000 7fc1b4ce4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:21:47.081 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.102+0000 7fc1b4ce4640 10 monclient(hunting): picked mon.noname-a con 0x7fc1b00b0770 addr v1:172.21.15.104:6789/0 2024-03-29T11:21:47.081 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.102+0000 7fc1b4ce4640 10 monclient(hunting): picked mon.noname-b con 0x7fc1b014a700 addr v1:172.21.15.104:6790/0 2024-03-29T11:21:47.081 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.102+0000 7fc1b4ce4640 10 monclient(hunting): picked mon.noname-c con 0x7fc1b00ac5a0 addr v1:172.21.15.119:6789/0 2024-03-29T11:21:47.081 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.102+0000 7fc1b4ce4640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1b00aba70 con 0x7fc1b00b0770 2024-03-29T11:21:47.081 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.102+0000 7fc1b4ce4640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1b00aecb0 con 0x7fc1b00ac5a0 2024-03-29T11:21:47.082 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.102+0000 7fc1b4ce4640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1b00af150 con 0x7fc1b014a700 2024-03-29T11:21:47.082 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.102+0000 7fc1b4ce4640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:47.082 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.102+0000 7fc1b4ce4640 10 monclient(hunting): authenticate will time out at 1088.353270s 2024-03-29T11:21:47.082 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.102+0000 7fc1add9b640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7fc1b014a700 0x7fc1b00abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:42448/0 (socket says 172.21.15.104:42448) 2024-03-29T11:21:47.082 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.102+0000 7fc1add9b640 1 -- 172.21.15.104:0/2106253746 learned_addr learned my addr 172.21.15.104:0/2106253746 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:21:47.082 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.103+0000 7fc1ad59a640 1 -- 172.21.15.104:0/2106253746 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 75962812 0 0) 0x7fc1b00af150 con 0x7fc1b014a700 2024-03-29T11:21:47.082 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.103+0000 7fc1ad59a640 10 monclient(hunting): _negotiate 2024-03-29T11:21:47.082 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.103+0000 7fc1ad59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:47.082 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.103+0000 7fc1ad59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:47.082 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.103+0000 7fc1ad59a640 10 monclient(hunting): my global_id is 4331 2024-03-29T11:21:47.082 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.103+0000 7fc1ad59a640 1 -- 172.21.15.104:0/2106253746 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc198003520 con 0x7fc1b014a700 2024-03-29T11:21:47.082 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.103+0000 7fc1ad59a640 1 -- 172.21.15.104:0/2106253746 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3881379044 0 0) 0x7fc1b00aecb0 con 0x7fc1b00ac5a0 2024-03-29T11:21:47.082 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.104+0000 7fc1ad59a640 10 monclient(hunting): _negotiate 2024-03-29T11:21:47.082 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.104+0000 7fc1ad59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:47.083 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.104+0000 7fc1ad59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:47.083 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.104+0000 7fc1ad59a640 10 monclient(hunting): my global_id is 4297 2024-03-29T11:21:47.083 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.104+0000 7fc1ad59a640 1 -- 172.21.15.104:0/2106253746 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1b00af150 con 0x7fc1b00ac5a0 2024-03-29T11:21:47.083 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.104+0000 7fc1ad59a640 1 -- 172.21.15.104:0/2106253746 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3428521256 0 0) 0x7fc1b00aba70 con 0x7fc1b00b0770 2024-03-29T11:21:47.083 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.104+0000 7fc1ad59a640 10 monclient(hunting): _negotiate 2024-03-29T11:21:47.083 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.104+0000 7fc1ad59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:47.083 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.105+0000 7fc1ad59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:47.083 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.105+0000 7fc1ad59a640 10 monclient(hunting): my global_id is 4320 2024-03-29T11:21:47.083 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.105+0000 7fc1ad59a640 1 -- 172.21.15.104:0/2106253746 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1b00aecb0 con 0x7fc1b00b0770 2024-03-29T11:21:47.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.105+0000 7fc1ad59a640 1 -- 172.21.15.104:0/2106253746 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2312658751 0 0) 0x7fc198003520 con 0x7fc1b014a700 2024-03-29T11:21:47.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.105+0000 7fc1ad59a640 1 -- 172.21.15.104:0/2106253746 >> v1:172.21.15.119:6789/0 conn(0x7fc1b00ac5a0 legacy=0x7fc1b0151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:47.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.105+0000 7fc1ad59a640 1 -- 172.21.15.104:0/2106253746 >> v1:172.21.15.104:6789/0 conn(0x7fc1b00b0770 legacy=0x7fc1b0149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:47.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.106+0000 7fc1ad59a640 10 monclient: _finish_hunting 0 2024-03-29T11:21:47.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.106+0000 7fc1ad59a640 1 monclient: found mon.noname-b 2024-03-29T11:21:47.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.106+0000 7fc1ad59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:47.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.106+0000 7fc1ad59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.104:6790/0 2024-03-29T11:21:47.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.106+0000 7fc1ad59a640 1 -- 172.21.15.104:0/2106253746 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b00ac9d0 con 0x7fc1b014a700 2024-03-29T11:21:47.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.106+0000 7fc1ad59a640 10 monclient: _finish_auth 0 2024-03-29T11:21:47.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.106+0000 7fc1ad59a640 10 monclient: _check_auth_tickets 2024-03-29T11:21:47.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.106+0000 7fc1ad59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:47.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.106+0000 7fc1ad59a640 1 -- 172.21.15.104:0/2106253746 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fc1a4001610 con 0x7fc1b014a700 2024-03-29T11:21:47.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.106+0000 7fc1ad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:47.088 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.106+0000 7fc1ad59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:47.088 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.106+0000 7fc1ad59a640 10 monclient: dump: 2024-03-29T11:21:47.088 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:21:47.088 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:21:47.088 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:47.088 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:47.088 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:47.088 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:21:47.088 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:21:47.088 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:21:47.088 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:21:47.088 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:21:47.088 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.106+0000 7fc1b4ce4640 5 monclient: authenticate success, global_id 4331 2024-03-29T11:21:47.088 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.106+0000 7fc1b4ce4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:47.088 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.106+0000 7fc1ad59a640 1 -- 172.21.15.104:0/2106253746 <== mon.2 v1:172.21.15.104:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc1a4001770 con 0x7fc1b014a700 2024-03-29T11:21:47.089 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.106+0000 7fc1ad59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:47.089 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.106+0000 7fc1ad59a640 1 -- 172.21.15.104:0/2106253746 <== mon.2 v1:172.21.15.104:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fc1a4001b80 con 0x7fc1b014a700 2024-03-29T11:21:47.089 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.106+0000 7fc1ad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:47.089 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.106+0000 7fc1ad59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:47.089 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.106+0000 7fc1ad59a640 10 monclient: dump: 2024-03-29T11:21:47.089 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:21:47.089 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:21:47.089 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:47.089 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:47.089 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:47.089 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:21:47.089 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:21:47.089 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:21:47.089 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:21:47.089 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:21:47.089 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.106+0000 7fc1b4ce4640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:47.090 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.106+0000 7fc1b4ce4640 4 set_mon_vals no callback set 2024-03-29T11:21:47.090 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.107+0000 7fc1b4ce4640 10 monclient: shutdown 2024-03-29T11:21:47.090 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.107+0000 7fc1b4ce4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:47.090 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.107+0000 7fc1b4ce4640 1 -- 172.21.15.104:0/2106253746 >> v1:172.21.15.104:6790/0 conn(0x7fc1b014a700 legacy=0x7fc1b00abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:47.090 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.108+0000 7fc1b4ce4640 1 -- 172.21.15.104:0/2106253746 shutdown_connections 2024-03-29T11:21:47.090 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.108+0000 7fc1b4ce4640 1 -- 172.21.15.104:0/2106253746 >> 172.21.15.104:0/2106253746 conn(0x7fc1b006b660 msgr2=0x7fc1b006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:47.090 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.108+0000 7fc1b4ce4640 1 -- 172.21.15.104:0/2106253746 shutdown_connections 2024-03-29T11:21:47.090 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.108+0000 7fc1b4ce4640 1 -- 172.21.15.104:0/2106253746 wait complete. 2024-03-29T11:21:47.090 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.109+0000 7fc1b4ce4640 10 monclient: build_initial_monmap 2024-03-29T11:21:47.090 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.109+0000 7fc1b4ce4640 10 monclient: monmap: 2024-03-29T11:21:47.090 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:21:47.090 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:47.090 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:21:47.090 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:21:47.091 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:47.091 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:21:47.091 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:21:47.091 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:21:47.091 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:21:47.091 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:21:47.091 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.110+0000 7fc1b4ce4640 1 librados: starting msgr at 2024-03-29T11:21:47.091 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.110+0000 7fc1b4ce4640 1 librados: starting objecter 2024-03-29T11:21:47.091 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.110+0000 7fc1b4ce4640 1 Processor -- start 2024-03-29T11:21:47.091 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.110+0000 7fc1b4ce4640 1 -- start start 2024-03-29T11:21:47.091 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.110+0000 7fc1b4ce4640 1 librados: setting wanted keys 2024-03-29T11:21:47.091 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.110+0000 7fc1b4ce4640 1 librados: calling monclient init 2024-03-29T11:21:47.091 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.110+0000 7fc1b4ce4640 10 monclient: init 2024-03-29T11:21:47.100 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.111+0000 7fc1b4ce4640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:47.100 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.111+0000 7fc1b4ce4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:21:47.100 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.111+0000 7fc1b4ce4640 10 monclient(hunting): picked mon.noname-b con 0x7fc1b00ac5a0 addr v1:172.21.15.119:6789/0 2024-03-29T11:21:47.100 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.112+0000 7fc1b4ce4640 10 monclient(hunting): picked mon.noname-c con 0x7fc1b00b0770 addr v1:172.21.15.104:6790/0 2024-03-29T11:21:47.101 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.112+0000 7fc1b4ce4640 10 monclient(hunting): picked mon.noname-a con 0x7fc1b014a700 addr v1:172.21.15.104:6789/0 2024-03-29T11:21:47.101 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.112+0000 7fc1b4ce4640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1b0128100 con 0x7fc1b014a700 2024-03-29T11:21:47.101 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.112+0000 7fc1b4ce4640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1b0145b90 con 0x7fc1b00ac5a0 2024-03-29T11:21:47.101 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.112+0000 7fc1add9b640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7fc1b00b0770 0x7fc1b0144f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:42766/0 (socket says 172.21.15.104:42766) 2024-03-29T11:21:47.101 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.112+0000 7fc1add9b640 1 -- 172.21.15.104:0/3945343530 learned_addr learned my addr 172.21.15.104:0/3945343530 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:21:47.101 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.112+0000 7fc1b4ce4640 1 -- 172.21.15.104:0/3945343530 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1b0125d80 con 0x7fc1b00b0770 2024-03-29T11:21:47.101 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.112+0000 7fc1b4ce4640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:47.101 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.112+0000 7fc1b4ce4640 10 monclient(hunting): authenticate will time out at 1088.363401s 2024-03-29T11:21:47.101 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.112+0000 7fc1ae59c640 1 --1- 172.21.15.104:0/3945343530 >> v1:172.21.15.119:6789/0 conn(0x7fc1b00ac5a0 0x7fc1b01267f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:59076/0 (socket says 172.21.15.104:59076) 2024-03-29T11:21:47.101 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.112+0000 7fc19effd640 10 client.?.objecter ms_handle_connect 0x7fc1b00b0770 2024-03-29T11:21:47.101 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.112+0000 7fc19effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:47.101 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.112+0000 7fc19effd640 10 client.?.objecter ms_handle_connect 0x7fc1b014a700 2024-03-29T11:21:47.101 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.113+0000 7fc19effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:47.101 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.113+0000 7fc19effd640 10 client.?.objecter ms_handle_connect 0x7fc1b00ac5a0 2024-03-29T11:21:47.101 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.113+0000 7fc19effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:47.102 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.113+0000 7fc19effd640 1 -- 172.21.15.104:0/3945343530 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1065325474 0 0) 0x7fc1b0125d80 con 0x7fc1b00b0770 2024-03-29T11:21:47.102 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.113+0000 7fc19effd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:47.102 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.113+0000 7fc19effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:47.102 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.113+0000 7fc19effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:47.102 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.113+0000 7fc19effd640 10 monclient(hunting): my global_id is 4334 2024-03-29T11:21:47.102 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.113+0000 7fc19effd640 1 -- 172.21.15.104:0/3945343530 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc190004eb0 con 0x7fc1b00b0770 2024-03-29T11:21:47.102 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.113+0000 7fc19effd640 1 -- 172.21.15.104:0/3945343530 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3426649366 0 0) 0x7fc1b0145b90 con 0x7fc1b00ac5a0 2024-03-29T11:21:47.102 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.113+0000 7fc19effd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:47.102 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.113+0000 7fc19effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:47.102 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.113+0000 7fc19effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:47.102 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.113+0000 7fc19effd640 10 monclient(hunting): my global_id is 4300 2024-03-29T11:21:47.102 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.113+0000 7fc19effd640 1 -- 172.21.15.104:0/3945343530 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1b0125d80 con 0x7fc1b00ac5a0 2024-03-29T11:21:47.102 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.113+0000 7fc19effd640 1 -- 172.21.15.104:0/3945343530 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1139481855 0 0) 0x7fc1b0128100 con 0x7fc1b014a700 2024-03-29T11:21:47.103 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.113+0000 7fc19effd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:47.103 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.114+0000 7fc19effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:47.103 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.114+0000 7fc19effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:47.103 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.114+0000 7fc19effd640 10 monclient(hunting): my global_id is 4323 2024-03-29T11:21:47.103 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.114+0000 7fc19effd640 1 -- 172.21.15.104:0/3945343530 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1b0145b90 con 0x7fc1b014a700 2024-03-29T11:21:47.103 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.114+0000 7fc19effd640 1 -- 172.21.15.104:0/3945343530 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 534513843 0 0) 0x7fc190004eb0 con 0x7fc1b00b0770 2024-03-29T11:21:47.103 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.114+0000 7fc19effd640 1 -- 172.21.15.104:0/3945343530 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc1b0128100 con 0x7fc1b00b0770 2024-03-29T11:21:47.103 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.114+0000 7fc19effd640 1 -- 172.21.15.104:0/3945343530 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fc1a4001bc0 con 0x7fc1b00b0770 2024-03-29T11:21:47.103 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.114+0000 7fc19effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:47.103 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.114+0000 7fc19effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:47.103 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.114+0000 7fc19effd640 10 monclient(hunting): dump: 2024-03-29T11:21:47.103 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:21:47.103 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:21:47.103 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:47.103 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:47.104 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:47.104 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:21:47.104 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:21:47.104 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:21:47.104 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:21:47.104 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:21:47.104 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.114+0000 7fc19effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:47.104 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.114+0000 7fc19effd640 1 -- 172.21.15.104:0/3945343530 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 686713073 0 0) 0x7fc1b0145b90 con 0x7fc1b014a700 2024-03-29T11:21:47.104 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.114+0000 7fc19effd640 1 -- 172.21.15.104:0/3945343530 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc190004eb0 con 0x7fc1b014a700 2024-03-29T11:21:47.104 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.115+0000 7fc19effd640 1 -- 172.21.15.104:0/3945343530 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3003085457 0 0) 0x7fc1b0125d80 con 0x7fc1b00ac5a0 2024-03-29T11:21:47.104 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.115+0000 7fc19effd640 1 -- 172.21.15.104:0/3945343530 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc1b0145b90 con 0x7fc1b00ac5a0 2024-03-29T11:21:47.104 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.115+0000 7fc19effd640 1 -- 172.21.15.104:0/3945343530 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fc1a8001430 con 0x7fc1b014a700 2024-03-29T11:21:47.104 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.115+0000 7fc19effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:47.104 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.115+0000 7fc19effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:47.105 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.115+0000 7fc19effd640 10 monclient(hunting): dump: 2024-03-29T11:21:47.105 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:21:47.105 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:21:47.105 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:47.105 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:47.105 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:47.105 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:21:47.105 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:21:47.105 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:21:47.105 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:21:47.105 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:21:47.105 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.115+0000 7fc19effd640 1 -- 172.21.15.104:0/3945343530 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fc1a0001910 con 0x7fc1b00ac5a0 2024-03-29T11:21:47.105 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.115+0000 7fc19effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:47.105 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.115+0000 7fc19effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:47.105 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.115+0000 7fc19effd640 10 monclient(hunting): dump: 2024-03-29T11:21:47.106 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:21:47.106 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:21:47.106 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:47.106 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:47.106 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:47.106 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:21:47.106 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:21:47.106 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:21:47.106 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:21:47.106 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:21:47.106 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.115+0000 7fc19effd640 1 -- 172.21.15.104:0/3945343530 <== mon.0 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2904533378 0 0) 0x7fc190004eb0 con 0x7fc1b014a700 2024-03-29T11:21:47.106 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.115+0000 7fc19effd640 1 -- 172.21.15.104:0/3945343530 >> v1:172.21.15.104:6790/0 conn(0x7fc1b00b0770 legacy=0x7fc1b0144f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:47.106 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.115+0000 7fc19effd640 1 -- 172.21.15.104:0/3945343530 >> v1:172.21.15.119:6789/0 conn(0x7fc1b00ac5a0 legacy=0x7fc1b01267f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:47.106 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.115+0000 7fc19effd640 10 monclient: _finish_hunting 0 2024-03-29T11:21:47.107 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.115+0000 7fc19effd640 1 monclient: found mon.a 2024-03-29T11:21:47.107 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.116+0000 7fc19effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:47.107 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.116+0000 7fc19effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:21:47.107 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.116+0000 7fc19effd640 1 -- 172.21.15.104:0/3945343530 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b0126f00 con 0x7fc1b014a700 2024-03-29T11:21:47.107 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.116+0000 7fc19effd640 10 monclient: _finish_auth 0 2024-03-29T11:21:47.107 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.116+0000 7fc19effd640 10 monclient: _check_auth_tickets 2024-03-29T11:21:47.107 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.116+0000 7fc19effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:47.107 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.116+0000 7fc1b4ce4640 5 monclient: authenticate success, global_id 4323 2024-03-29T11:21:47.107 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.116+0000 7fc1b4ce4640 10 monclient: _renew_subs 2024-03-29T11:21:47.107 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.116+0000 7fc1b4ce4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:21:47.107 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.116+0000 7fc1b4ce4640 1 -- 172.21.15.104:0/3945343530 --> v1:172.21.15.104:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1b0145620 con 0x7fc1b014a700 2024-03-29T11:21:47.107 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.116+0000 7fc1b4ce4640 10 client.4323.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:47.107 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.116+0000 7fc1b4ce4640 10 monclient: _renew_subs 2024-03-29T11:21:47.107 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.116+0000 7fc1b4ce4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:21:47.107 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.116+0000 7fc1b4ce4640 1 -- 172.21.15.104:0/3945343530 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1b01df3c0 con 0x7fc1b014a700 2024-03-29T11:21:47.108 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.116+0000 7fc1b4ce4640 1 librados: init done 2024-03-29T11:21:47.108 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.116+0000 7fc19effd640 1 -- 172.21.15.104:0/3945343530 <== mon.0 v1:172.21.15.104:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc1a8001940 con 0x7fc1b014a700 2024-03-29T11:21:47.108 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.116+0000 7fc19effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:47.108 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.116+0000 7fc19effd640 1 -- 172.21.15.104:0/3945343530 <== mon.0 v1:172.21.15.104:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fc1a8003b10 con 0x7fc1b014a700 2024-03-29T11:21:47.108 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.116+0000 7fc19ffff640 4 set_mon_vals no callback set 2024-03-29T11:21:47.108 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.116+0000 7fc19effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:47.108 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.116+0000 7fc19effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:47.108 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.117+0000 7fc19effd640 10 monclient: dump: 2024-03-29T11:21:47.108 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:21:47.108 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:21:47.108 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:47.108 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:47.108 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:47.108 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:21:47.108 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:21:47.109 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:21:47.109 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:21:47.109 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:21:47.109 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.117+0000 7fc1b4ce4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:47.109 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.117+0000 7fc1b4ce4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:47.109 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.118+0000 7fc1b4ce4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:21:47.109 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.118+0000 7fc1b4ce4640 1 -- 172.21.15.104:0/3945343530 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc194005080 con 0x7fc1b014a700 2024-03-29T11:21:47.109 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.118+0000 7fc19effd640 1 -- 172.21.15.104:0/3945343530 <== mon.0 v1:172.21.15.104:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7fc1a8019ae0 con 0x7fc1b014a700 2024-03-29T11:21:47.109 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.120+0000 7fc19effd640 1 -- 172.21.15.104:0/3945343530 <== mon.0 v1:172.21.15.104:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7fc1a8083830 con 0x7fc1b014a700 2024-03-29T11:21:47.109 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.120+0000 7fc19effd640 10 client.4323.objecter ms_dispatch 0x7fc1b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:47.109 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.120+0000 7fc19effd640 3 client.4323.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:47.109 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.120+0000 7fc19effd640 3 client.4323.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:47.109 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.120+0000 7fc19effd640 20 client.4323.objecter dump_active .. 0 homeless 2024-03-29T11:21:47.109 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.120+0000 7fc1ae59c640 10 monclient: get_auth_request con 0x7fc19006e370 auth_method 0 2024-03-29T11:21:47.109 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.121+0000 7fc19effd640 10 client.4323.objecter ms_handle_connect 0x7fc19006e370 2024-03-29T11:21:47.110 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.128+0000 7fc19effd640 1 -- 172.21.15.104:0/3945343530 <== mon.0 v1:172.21.15.104:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc1a804f110 con 0x7fc1b014a700 2024-03-29T11:21:47.110 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.128+0000 7fc19effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:47.110 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.128+0000 7fc19effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:47.476 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.498+0000 7fc1b4ce4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:47.476 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.498+0000 7fc1b4ce4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:47.476 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.498+0000 7fc1b4ce4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:21:47.476 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.498+0000 7fc1b4ce4640 1 -- 172.21.15.104:0/3945343530 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc194005330 con 0x7fc1b014a700 2024-03-29T11:21:47.476 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.499+0000 7fc19effd640 1 -- 172.21.15.104:0/3945343530 <== mon.0 v1:172.21.15.104:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7fc1a8051040 con 0x7fc1b014a700 2024-03-29T11:21:47.476 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.499+0000 7fc19effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:47.476 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.499+0000 7fc19effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:47.476 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:21:47.477 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:47.479 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.502+0000 7fc1b4ce4640 10 librados: watch_flush enter 2024-03-29T11:21:47.479 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.502+0000 7fc1b4ce4640 10 librados: watch_flush exit 2024-03-29T11:21:47.479 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.502+0000 7fc1b4ce4640 20 client.4323.objecter shutdown clearing up homeless session... 2024-03-29T11:21:47.479 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.502+0000 7fc1b4ce4640 10 client.4323.objecter successfully canceled tick 2024-03-29T11:21:47.480 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.502+0000 7fc1b4ce4640 1 -- 172.21.15.104:0/3945343530 >> v1:172.21.15.104:6816/3341126660 conn(0x7fc19006e370 legacy=0x7fc190070830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:47.480 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.502+0000 7fc1b4ce4640 10 monclient: shutdown 2024-03-29T11:21:47.480 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.502+0000 7fc1b4ce4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:47.480 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.502+0000 7fc1b4ce4640 1 -- 172.21.15.104:0/3945343530 >> v1:172.21.15.104:6789/0 conn(0x7fc1b014a700 legacy=0x7fc1b01decb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:47.480 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.502+0000 7fc1b4ce4640 1 -- 172.21.15.104:0/3945343530 shutdown_connections 2024-03-29T11:21:47.480 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.502+0000 7fc1b4ce4640 1 -- 172.21.15.104:0/3945343530 >> 172.21.15.104:0/3945343530 conn(0x7fc1b006b660 msgr2=0x7fc1b00b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:47.480 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.503+0000 7fc1b4ce4640 1 -- 172.21.15.104:0/3945343530 shutdown_connections 2024-03-29T11:21:47.480 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.503+0000 7fc1b4ce4640 1 -- 172.21.15.104:0/3945343530 wait complete. 2024-03-29T11:21:47.480 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:47.503+0000 7fc1b4ce4640 1 librados: shutdown 2024-03-29T11:21:47.493 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:54.300 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:54.449 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.473+0000 7f16f7fff640 10 monclient: get_monmap_and_config 2024-03-29T11:21:54.449 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.473+0000 7f16f7fff640 10 monclient: build_initial_monmap 2024-03-29T11:21:54.449 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.473+0000 7f16f7fff640 10 monclient: monmap: 2024-03-29T11:21:54.449 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:21:54.450 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:21:54.450 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:21:54.474081+0000 2024-03-29T11:21:54.450 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:21:54.474081+0000 2024-03-29T11:21:54.450 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:54.450 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:21:54.450 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:21:54.450 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:21:54.450 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:21:54.450 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:21:54.451 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.475+0000 7f16f7fff640 1 Processor -- start 2024-03-29T11:21:54.452 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.476+0000 7f16f7fff640 1 -- start start 2024-03-29T11:21:54.452 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.476+0000 7f16f7fff640 10 monclient: init 2024-03-29T11:21:54.453 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.476+0000 7f16f7fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:54.453 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.477+0000 7f16f7fff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:21:54.454 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.477+0000 7f16f7fff640 10 monclient(hunting): picked mon.noname-b con 0x7f16f00345e0 addr v1:172.21.15.104:6790/0 2024-03-29T11:21:54.454 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.477+0000 7f16f7fff640 10 monclient(hunting): picked mon.noname-a con 0x7f16f00349c0 addr v1:172.21.15.104:6789/0 2024-03-29T11:21:54.454 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.477+0000 7f16f7fff640 10 monclient(hunting): picked mon.noname-c con 0x7f16f002b570 addr v1:172.21.15.119:6789/0 2024-03-29T11:21:54.454 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.477+0000 7f16f7fff640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f0032ef0 con 0x7f16f00349c0 2024-03-29T11:21:54.454 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.477+0000 7f16f7fff640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f00be170 con 0x7f16f002b570 2024-03-29T11:21:54.454 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.477+0000 7f16f7fff640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f002c720 con 0x7f16f00345e0 2024-03-29T11:21:54.454 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.477+0000 7f16f7fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:54.454 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.477+0000 7f16f7fff640 10 monclient(hunting): authenticate will time out at 1095.728230s 2024-03-29T11:21:54.454 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.477+0000 7f16f77fe640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f16f002b570 0x7f16f00c7970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:51636/0 (socket says 172.21.15.104:51636) 2024-03-29T11:21:54.454 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.477+0000 7f16f77fe640 1 -- 172.21.15.104:0/6315961 learned_addr learned my addr 172.21.15.104:0/6315961 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:21:54.455 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.477+0000 7f16f5ffb640 1 -- 172.21.15.104:0/6315961 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2265657393 0 0) 0x7f16f00be170 con 0x7f16f002b570 2024-03-29T11:21:54.455 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.478+0000 7f16f5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:54.455 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.478+0000 7f16f5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:54.455 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.478+0000 7f16f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:54.455 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.478+0000 7f16f5ffb640 10 monclient(hunting): my global_id is 4303 2024-03-29T11:21:54.455 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.478+0000 7f16f5ffb640 1 -- 172.21.15.104:0/6315961 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16e0003520 con 0x7f16f002b570 2024-03-29T11:21:54.457 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.480+0000 7f16f5ffb640 1 -- 172.21.15.104:0/6315961 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3180258078 0 0) 0x7f16e0003520 con 0x7f16f002b570 2024-03-29T11:21:54.457 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.480+0000 7f16f5ffb640 1 -- 172.21.15.104:0/6315961 >> v1:172.21.15.104:6790/0 conn(0x7f16f00345e0 legacy=0x7f16f00bda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:54.457 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.480+0000 7f16f5ffb640 1 -- 172.21.15.104:0/6315961 >> v1:172.21.15.104:6789/0 conn(0x7f16f00349c0 legacy=0x7f16f00c7260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:21:54.457 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.480+0000 7f16f5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:21:54.457 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.480+0000 7f16f5ffb640 1 monclient: found mon.noname-c 2024-03-29T11:21:54.458 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.480+0000 7f16f5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:54.458 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.480+0000 7f16f5ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.119:6789/0 2024-03-29T11:21:54.458 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.480+0000 7f16f5ffb640 1 -- 172.21.15.104:0/6315961 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f002c990 con 0x7f16f002b570 2024-03-29T11:21:54.458 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.480+0000 7f16f5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:21:54.458 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.480+0000 7f16f5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:21:54.458 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.480+0000 7f16f5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:54.458 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.480+0000 7f16f5ffb640 1 -- 172.21.15.104:0/6315961 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f16e8001690 con 0x7f16f002b570 2024-03-29T11:21:54.458 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.480+0000 7f16f5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:54.458 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.480+0000 7f16f5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:54.458 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.480+0000 7f16f5ffb640 10 monclient: dump: 2024-03-29T11:21:54.458 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:21:54.458 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:21:54.458 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:54.458 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:54.459 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:54.459 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:21:54.459 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:21:54.459 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:21:54.459 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:21:54.459 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:21:54.459 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.480+0000 7f16f7fff640 5 monclient: authenticate success, global_id 4303 2024-03-29T11:21:54.459 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.480+0000 7f16f7fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:54.463 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.481+0000 7f16f5ffb640 1 -- 172.21.15.104:0/6315961 <== mon.1 v1:172.21.15.119:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f16e8001860 con 0x7f16f002b570 2024-03-29T11:21:54.463 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.481+0000 7f16f5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:54.463 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.481+0000 7f16f5ffb640 1 -- 172.21.15.104:0/6315961 <== mon.1 v1:172.21.15.119:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f16e8001cb0 con 0x7f16f002b570 2024-03-29T11:21:54.463 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.481+0000 7f16f5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:54.464 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.481+0000 7f16f5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:54.464 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.481+0000 7f16f5ffb640 10 monclient: dump: 2024-03-29T11:21:54.464 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:21:54.464 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:21:54.464 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:54.464 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:54.464 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:54.464 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:21:54.464 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:21:54.464 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:21:54.464 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:21:54.464 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:21:54.464 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.481+0000 7f16f7fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:54.464 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.481+0000 7f16f7fff640 4 set_mon_vals no callback set 2024-03-29T11:21:54.465 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.482+0000 7f16f7fff640 10 monclient: shutdown 2024-03-29T11:21:54.465 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.482+0000 7f16f7fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:54.465 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.482+0000 7f16f7fff640 1 -- 172.21.15.104:0/6315961 >> v1:172.21.15.119:6789/0 conn(0x7f16f002b570 legacy=0x7f16f00c7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:54.465 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.482+0000 7f16f7fff640 1 -- 172.21.15.104:0/6315961 shutdown_connections 2024-03-29T11:21:54.465 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.482+0000 7f16f7fff640 1 -- 172.21.15.104:0/6315961 >> 172.21.15.104:0/6315961 conn(0x7f16f00058f0 msgr2=0x7f16f0005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:54.465 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.483+0000 7f16f7fff640 1 -- 172.21.15.104:0/6315961 shutdown_connections 2024-03-29T11:21:54.465 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.483+0000 7f16f7fff640 1 -- 172.21.15.104:0/6315961 wait complete. 2024-03-29T11:21:54.465 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.484+0000 7f16f7fff640 10 monclient: build_initial_monmap 2024-03-29T11:21:54.465 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.484+0000 7f16f7fff640 10 monclient: monmap: 2024-03-29T11:21:54.465 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:21:54.465 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:54.465 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:21:54.465 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:21:54.465 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:54.465 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:21:54.466 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:21:54.466 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:21:54.466 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:21:54.466 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:21:54.466 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.485+0000 7f16f7fff640 1 librados: starting msgr at 2024-03-29T11:21:54.466 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.485+0000 7f16f7fff640 1 librados: starting objecter 2024-03-29T11:21:54.466 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.485+0000 7f16f7fff640 1 Processor -- start 2024-03-29T11:21:54.466 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.485+0000 7f16f7fff640 1 -- start start 2024-03-29T11:21:54.466 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.486+0000 7f16f7fff640 1 librados: setting wanted keys 2024-03-29T11:21:54.466 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.486+0000 7f16f7fff640 1 librados: calling monclient init 2024-03-29T11:21:54.466 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.486+0000 7f16f7fff640 10 monclient: init 2024-03-29T11:21:54.466 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.486+0000 7f16f7fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:54.466 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.487+0000 7f16f7fff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:54.466 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.487+0000 7f16f7fff640 10 monclient(hunting): picked mon.noname-c con 0x7f16f002b570 addr v1:172.21.15.104:6790/0 2024-03-29T11:21:54.466 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.487+0000 7f16f7fff640 10 monclient(hunting): picked mon.noname-b con 0x7f16f00c0350 addr v1:172.21.15.119:6789/0 2024-03-29T11:21:54.467 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.490+0000 7f16f67fc640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f16f00c0350 0x7f16f00b39c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:51956/0 (socket says 172.21.15.104:51956) 2024-03-29T11:21:54.467 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.490+0000 7f16f77fe640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f16f00c4860 0x7f16f00b40d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:46375/0 (socket says 172.21.15.104:46375) 2024-03-29T11:21:54.467 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.490+0000 7f16f67fc640 1 -- 172.21.15.104:0/1983216349 learned_addr learned my addr 172.21.15.104:0/1983216349 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:21:54.467 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.490+0000 7f16f7fff640 10 monclient(hunting): picked mon.noname-a con 0x7f16f00c4860 addr v1:172.21.15.104:6789/0 2024-03-29T11:21:54.473 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.490+0000 7f16dffff640 10 client.?.objecter ms_handle_connect 0x7f16f00c0350 2024-03-29T11:21:54.473 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.490+0000 7f16dffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:54.473 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.490+0000 7f16dffff640 10 client.?.objecter ms_handle_connect 0x7f16f00c4860 2024-03-29T11:21:54.473 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.490+0000 7f16dffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:54.473 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.491+0000 7f16f7fff640 1 -- 172.21.15.104:0/1983216349 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f002e040 con 0x7f16f00c4860 2024-03-29T11:21:54.473 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.491+0000 7f16f7fff640 1 -- 172.21.15.104:0/1983216349 --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f00b47e0 con 0x7f16f00c0350 2024-03-29T11:21:54.473 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.491+0000 7f16f7fff640 1 -- 172.21.15.104:0/1983216349 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f002e290 con 0x7f16f002b570 2024-03-29T11:21:54.473 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.491+0000 7f16f7fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:54.473 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.491+0000 7f16f7fff640 10 monclient(hunting): authenticate will time out at 1095.742945s 2024-03-29T11:21:54.474 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.492+0000 7f16dffff640 10 client.?.objecter ms_handle_connect 0x7f16f002b570 2024-03-29T11:21:54.474 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.492+0000 7f16dffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:54.474 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.492+0000 7f16dffff640 1 -- 172.21.15.104:0/1983216349 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 139697589 0 0) 0x7f16f002e040 con 0x7f16f00c4860 2024-03-29T11:21:54.474 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.492+0000 7f16dffff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:54.474 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.492+0000 7f16dffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:54.474 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.492+0000 7f16dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:54.474 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.492+0000 7f16dffff640 10 monclient(hunting): my global_id is 4326 2024-03-29T11:21:54.474 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.492+0000 7f16dffff640 1 -- 172.21.15.104:0/1983216349 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16e4004eb0 con 0x7f16f00c4860 2024-03-29T11:21:54.474 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.492+0000 7f16dffff640 1 -- 172.21.15.104:0/1983216349 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 719503279 0 0) 0x7f16f002e290 con 0x7f16f002b570 2024-03-29T11:21:54.474 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.492+0000 7f16dffff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:54.474 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.492+0000 7f16dffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:54.475 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.493+0000 7f16dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:54.475 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.493+0000 7f16dffff640 10 monclient(hunting): my global_id is 4337 2024-03-29T11:21:54.475 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.493+0000 7f16dffff640 1 -- 172.21.15.104:0/1983216349 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16f002e040 con 0x7f16f002b570 2024-03-29T11:21:54.475 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.493+0000 7f16dffff640 1 -- 172.21.15.104:0/1983216349 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3363187250 0 0) 0x7f16e4004eb0 con 0x7f16f00c4860 2024-03-29T11:21:54.475 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.493+0000 7f16dffff640 1 -- 172.21.15.104:0/1983216349 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f16f002e290 con 0x7f16f00c4860 2024-03-29T11:21:54.475 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.493+0000 7f16dffff640 1 -- 172.21.15.104:0/1983216349 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f16e8001b70 con 0x7f16f00c4860 2024-03-29T11:21:54.475 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.493+0000 7f16dffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:54.475 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.493+0000 7f16dffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:54.475 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.493+0000 7f16dffff640 10 monclient(hunting): dump: 2024-03-29T11:21:54.475 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:21:54.475 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:21:54.475 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:54.476 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:54.476 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:54.476 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:21:54.476 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:21:54.476 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:21:54.476 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:21:54.476 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:21:54.476 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.493+0000 7f16dffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:54.476 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.493+0000 7f16dffff640 1 -- 172.21.15.104:0/1983216349 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 401834007 0 0) 0x7f16f002e040 con 0x7f16f002b570 2024-03-29T11:21:54.476 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.493+0000 7f16dffff640 1 -- 172.21.15.104:0/1983216349 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f16e4004eb0 con 0x7f16f002b570 2024-03-29T11:21:54.476 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.493+0000 7f16dffff640 1 -- 172.21.15.104:0/1983216349 <== mon.0 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1364392813 0 0) 0x7f16f002e290 con 0x7f16f00c4860 2024-03-29T11:21:54.477 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.493+0000 7f16dffff640 1 -- 172.21.15.104:0/1983216349 >> v1:172.21.15.104:6790/0 conn(0x7f16f002b570 legacy=0x7f16f015cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:54.477 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.493+0000 7f16dffff640 1 -- 172.21.15.104:0/1983216349 >> v1:172.21.15.119:6789/0 conn(0x7f16f00c0350 legacy=0x7f16f00b39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:54.477 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.493+0000 7f16dffff640 10 monclient: _finish_hunting 0 2024-03-29T11:21:54.477 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.493+0000 7f16dffff640 1 monclient: found mon.a 2024-03-29T11:21:54.477 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.493+0000 7f16dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:54.477 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.493+0000 7f16dffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:21:54.477 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.493+0000 7f16dffff640 1 -- 172.21.15.104:0/1983216349 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f00c1190 con 0x7f16f00c4860 2024-03-29T11:21:54.477 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.493+0000 7f16dffff640 10 monclient: _finish_auth 0 2024-03-29T11:21:54.477 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.493+0000 7f16dffff640 10 monclient: _check_auth_tickets 2024-03-29T11:21:54.477 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.493+0000 7f16dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:54.477 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.493+0000 7f16f7fff640 5 monclient: authenticate success, global_id 4326 2024-03-29T11:21:54.477 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.493+0000 7f16f7fff640 10 monclient: _renew_subs 2024-03-29T11:21:54.477 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.493+0000 7f16f7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:21:54.477 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.493+0000 7f16f7fff640 1 -- 172.21.15.104:0/1983216349 --> v1:172.21.15.104:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16f00c0df0 con 0x7f16f00c4860 2024-03-29T11:21:54.477 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.493+0000 7f16f7fff640 10 client.4326.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:54.478 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.493+0000 7f16f7fff640 10 monclient: _renew_subs 2024-03-29T11:21:54.478 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.493+0000 7f16f7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:21:54.478 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.493+0000 7f16f7fff640 1 -- 172.21.15.104:0/1983216349 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f16f015d5d0 con 0x7f16f00c4860 2024-03-29T11:21:54.478 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.493+0000 7f16f7fff640 1 librados: init done 2024-03-29T11:21:54.478 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.494+0000 7f16dffff640 1 -- 172.21.15.104:0/1983216349 <== mon.0 v1:172.21.15.104:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f16e8001010 con 0x7f16f00c4860 2024-03-29T11:21:54.478 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.494+0000 7f16dffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:54.478 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.494+0000 7f16dffff640 1 -- 172.21.15.104:0/1983216349 <== mon.0 v1:172.21.15.104:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f16e8004d90 con 0x7f16f00c4860 2024-03-29T11:21:54.478 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.494+0000 7f16dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:54.478 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.494+0000 7f16dffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:54.478 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.494+0000 7f16dffff640 10 monclient: dump: 2024-03-29T11:21:54.478 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:21:54.478 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:21:54.478 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:54.478 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:21:54.479 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:54.479 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:21:54.479 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:21:54.479 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:21:54.479 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:21:54.479 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:21:54.479 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.494+0000 7f16f57fa640 4 set_mon_vals no callback set 2024-03-29T11:21:54.479 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.495+0000 7f16f7fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:54.479 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.495+0000 7f16f7fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:54.479 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.496+0000 7f16f7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:21:54.479 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.496+0000 7f16f7fff640 1 -- 172.21.15.104:0/1983216349 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16d8005080 con 0x7f16f00c4860 2024-03-29T11:21:54.480 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.499+0000 7f16dffff640 1 -- 172.21.15.104:0/1983216349 <== mon.0 v1:172.21.15.104:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f16e801ad60 con 0x7f16f00c4860 2024-03-29T11:21:54.480 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.502+0000 7f16dffff640 1 -- 172.21.15.104:0/1983216349 <== mon.0 v1:172.21.15.104:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f16e8084aa0 con 0x7f16f00c4860 2024-03-29T11:21:54.480 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.503+0000 7f16f6ffd640 10 monclient: get_auth_request con 0x7f16e406dd30 auth_method 0 2024-03-29T11:21:54.480 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.503+0000 7f16dffff640 10 client.4326.objecter ms_dispatch 0x7f16f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:54.480 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.503+0000 7f16dffff640 3 client.4326.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:54.481 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.503+0000 7f16dffff640 3 client.4326.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:54.481 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.503+0000 7f16dffff640 20 client.4326.objecter dump_active .. 0 homeless 2024-03-29T11:21:54.481 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.503+0000 7f16dffff640 10 client.4326.objecter ms_handle_connect 0x7f16e406dd30 2024-03-29T11:21:54.486 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.509+0000 7f16dffff640 1 -- 172.21.15.104:0/1983216349 <== mon.0 v1:172.21.15.104:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f16e8050380 con 0x7f16f00c4860 2024-03-29T11:21:54.486 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.509+0000 7f16dffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:54.486 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.509+0000 7f16dffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:54.808 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.832+0000 7f16f7fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:54.808 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.832+0000 7f16f7fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:54.808 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.832+0000 7f16f7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:21:54.808 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.832+0000 7f16f7fff640 1 -- 172.21.15.104:0/1983216349 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f16d8005330 con 0x7f16f00c4860 2024-03-29T11:21:54.809 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.832+0000 7f16dffff640 1 -- 172.21.15.104:0/1983216349 <== mon.0 v1:172.21.15.104:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f16e80522b0 con 0x7f16f00c4860 2024-03-29T11:21:54.809 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.832+0000 7f16dffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:54.809 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.832+0000 7f16dffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:54.812 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:21:54.812 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:54.817 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.838+0000 7f16f7fff640 10 librados: watch_flush enter 2024-03-29T11:21:54.818 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.840+0000 7f16f7fff640 10 librados: watch_flush exit 2024-03-29T11:21:54.818 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.840+0000 7f16f7fff640 20 client.4326.objecter shutdown clearing up homeless session... 2024-03-29T11:21:54.818 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.840+0000 7f16f7fff640 10 client.4326.objecter successfully canceled tick 2024-03-29T11:21:54.818 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.840+0000 7f16f7fff640 1 -- 172.21.15.104:0/1983216349 >> v1:172.21.15.104:6816/3341126660 conn(0x7f16e406dd30 legacy=0x7f16e40701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:54.818 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.841+0000 7f16f7fff640 10 monclient: shutdown 2024-03-29T11:21:54.818 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.841+0000 7f16f7fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:54.818 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.841+0000 7f16f7fff640 1 -- 172.21.15.104:0/1983216349 >> v1:172.21.15.104:6789/0 conn(0x7f16f00c4860 legacy=0x7f16f00b40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:54.818 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.841+0000 7f16f7fff640 1 -- 172.21.15.104:0/1983216349 shutdown_connections 2024-03-29T11:21:54.818 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.841+0000 7f16f7fff640 1 -- 172.21.15.104:0/1983216349 >> 172.21.15.104:0/1983216349 conn(0x7f16f00058f0 msgr2=0x7f16f0033d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:54.818 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.841+0000 7f16f7fff640 1 -- 172.21.15.104:0/1983216349 shutdown_connections 2024-03-29T11:21:54.818 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.841+0000 7f16f7fff640 1 -- 172.21.15.104:0/1983216349 wait complete. 2024-03-29T11:21:54.818 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:21:54.841+0000 7f16f7fff640 1 librados: shutdown 2024-03-29T11:21:54.833 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:01.640 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:01.786 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.810+0000 7f8354827640 10 monclient: get_monmap_and_config 2024-03-29T11:22:01.786 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.810+0000 7f8354827640 10 monclient: build_initial_monmap 2024-03-29T11:22:01.786 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.810+0000 7f8354827640 10 monclient: monmap: 2024-03-29T11:22:01.786 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:22:01.786 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:22:01.786 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:22:01.810954+0000 2024-03-29T11:22:01.786 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:22:01.810954+0000 2024-03-29T11:22:01.787 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:01.787 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:01.787 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:22:01.787 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:22:01.787 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:22:01.787 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:01.787 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.811+0000 7f8354827640 1 Processor -- start 2024-03-29T11:22:01.788 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.811+0000 7f8354827640 1 -- start start 2024-03-29T11:22:01.788 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.811+0000 7f8354827640 10 monclient: init 2024-03-29T11:22:01.789 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.812+0000 7f8354827640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:01.789 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.812+0000 7f8354827640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:22:01.789 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.812+0000 7f8354827640 10 monclient(hunting): picked mon.noname-c con 0x7f83501457f0 addr v1:172.21.15.119:6789/0 2024-03-29T11:22:01.789 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.812+0000 7f8354827640 10 monclient(hunting): picked mon.noname-a con 0x7f835013f650 addr v1:172.21.15.104:6789/0 2024-03-29T11:22:01.789 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.812+0000 7f8354827640 10 monclient(hunting): picked mon.noname-b con 0x7f83500a9d70 addr v1:172.21.15.104:6790/0 2024-03-29T11:22:01.790 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.812+0000 7f8354827640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83500aaef0 con 0x7f835013f650 2024-03-29T11:22:01.790 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.812+0000 7f8354827640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8350145bd0 con 0x7f83501457f0 2024-03-29T11:22:01.790 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.812+0000 7f8354827640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8350145eb0 con 0x7f83500a9d70 2024-03-29T11:22:01.790 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.812+0000 7f8354827640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:01.790 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.812+0000 7f8354827640 10 monclient(hunting): authenticate will time out at 1103.063575s 2024-03-29T11:22:01.790 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.813+0000 7f834e59c640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f83501457f0 0x7f835013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:32838/0 (socket says 172.21.15.104:32838) 2024-03-29T11:22:01.790 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.813+0000 7f834e59c640 1 -- 172.21.15.104:0/3542419099 learned_addr learned my addr 172.21.15.104:0/3542419099 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:22:01.791 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.815+0000 7f834d59a640 1 -- 172.21.15.104:0/3542419099 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1428202931 0 0) 0x7f8350145bd0 con 0x7f83501457f0 2024-03-29T11:22:01.791 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.815+0000 7f834d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:22:01.792 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.815+0000 7f834d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:01.792 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.815+0000 7f834d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:01.792 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.815+0000 7f834d59a640 10 monclient(hunting): my global_id is 4306 2024-03-29T11:22:01.792 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.815+0000 7f834d59a640 1 -- 172.21.15.104:0/3542419099 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f833c003520 con 0x7f83501457f0 2024-03-29T11:22:01.795 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.816+0000 7f834d59a640 1 -- 172.21.15.104:0/3542419099 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3089700198 0 0) 0x7f833c003520 con 0x7f83501457f0 2024-03-29T11:22:01.795 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.816+0000 7f834d59a640 1 -- 172.21.15.104:0/3542419099 >> v1:172.21.15.104:6790/0 conn(0x7f83500a9d70 legacy=0x7f835014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:01.795 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.816+0000 7f834d59a640 1 -- 172.21.15.104:0/3542419099 >> v1:172.21.15.104:6789/0 conn(0x7f835013f650 legacy=0x7f83500b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:01.795 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.816+0000 7f834d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:22:01.795 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.816+0000 7f834d59a640 1 monclient: found mon.noname-c 2024-03-29T11:22:01.795 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.816+0000 7f834d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:01.795 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.816+0000 7f834d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.119:6789/0 2024-03-29T11:22:01.795 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.816+0000 7f834d59a640 1 -- 172.21.15.104:0/3542419099 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8350146340 con 0x7f83501457f0 2024-03-29T11:22:01.795 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.816+0000 7f834d59a640 10 monclient: _finish_auth 0 2024-03-29T11:22:01.796 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.816+0000 7f834d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:22:01.796 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.816+0000 7f834d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:01.796 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.816+0000 7f8354827640 5 monclient: authenticate success, global_id 4306 2024-03-29T11:22:01.796 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.816+0000 7f8354827640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:01.796 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.817+0000 7f834d59a640 1 -- 172.21.15.104:0/3542419099 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f8348001690 con 0x7f83501457f0 2024-03-29T11:22:01.796 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.817+0000 7f834d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:01.796 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.817+0000 7f834d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:01.796 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.817+0000 7f834d59a640 10 monclient: dump: 2024-03-29T11:22:01.796 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:22:01.796 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:22:01.796 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:01.796 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:01.796 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:01.796 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:01.796 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:22:01.797 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:22:01.797 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:22:01.797 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:01.797 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.817+0000 7f8354827640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:01.797 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.817+0000 7f834d59a640 1 -- 172.21.15.104:0/3542419099 <== mon.1 v1:172.21.15.119:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f83480018f0 con 0x7f83501457f0 2024-03-29T11:22:01.797 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.817+0000 7f834d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:01.797 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.817+0000 7f8354827640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:01.797 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.817+0000 7f8354827640 4 set_mon_vals no callback set 2024-03-29T11:22:01.797 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.817+0000 7f834d59a640 1 -- 172.21.15.104:0/3542419099 <== mon.1 v1:172.21.15.119:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f8348001d00 con 0x7f83501457f0 2024-03-29T11:22:01.797 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.817+0000 7f834d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:01.797 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.817+0000 7f834d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:01.797 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.817+0000 7f834d59a640 10 monclient: dump: 2024-03-29T11:22:01.797 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:22:01.797 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:22:01.798 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:01.798 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:01.798 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:01.798 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:01.798 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:22:01.798 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:22:01.798 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:22:01.798 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:01.798 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.818+0000 7f8354827640 10 monclient: shutdown 2024-03-29T11:22:01.798 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.818+0000 7f8354827640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:01.798 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.818+0000 7f8354827640 1 -- 172.21.15.104:0/3542419099 >> v1:172.21.15.119:6789/0 conn(0x7f83501457f0 legacy=0x7f835013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:01.799 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.818+0000 7f8354827640 1 -- 172.21.15.104:0/3542419099 shutdown_connections 2024-03-29T11:22:01.799 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.818+0000 7f8354827640 1 -- 172.21.15.104:0/3542419099 >> 172.21.15.104:0/3542419099 conn(0x7f8350066a00 msgr2=0x7f8350066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:01.802 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.819+0000 7f8354827640 1 -- 172.21.15.104:0/3542419099 shutdown_connections 2024-03-29T11:22:01.802 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.819+0000 7f8354827640 1 -- 172.21.15.104:0/3542419099 wait complete. 2024-03-29T11:22:01.802 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.820+0000 7f8354827640 10 monclient: build_initial_monmap 2024-03-29T11:22:01.802 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.820+0000 7f8354827640 10 monclient: monmap: 2024-03-29T11:22:01.802 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:22:01.802 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:01.802 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:22:01.802 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:22:01.802 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:01.803 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:01.803 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:22:01.803 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:22:01.803 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:22:01.803 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:01.803 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.821+0000 7f8354827640 1 librados: starting msgr at 2024-03-29T11:22:01.803 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.821+0000 7f8354827640 1 librados: starting objecter 2024-03-29T11:22:01.803 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.821+0000 7f8354827640 1 Processor -- start 2024-03-29T11:22:01.803 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.821+0000 7f8354827640 1 -- start start 2024-03-29T11:22:01.803 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.821+0000 7f8354827640 1 librados: setting wanted keys 2024-03-29T11:22:01.803 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.821+0000 7f8354827640 1 librados: calling monclient init 2024-03-29T11:22:01.803 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.821+0000 7f8354827640 10 monclient: init 2024-03-29T11:22:01.803 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.822+0000 7f8354827640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:01.803 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.823+0000 7f8354827640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:22:01.804 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.823+0000 7f8354827640 10 monclient(hunting): picked mon.noname-a con 0x7f83500a9d70 addr v1:172.21.15.104:6789/0 2024-03-29T11:22:01.804 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.823+0000 7f8354827640 10 monclient(hunting): picked mon.noname-b con 0x7f835013f650 addr v1:172.21.15.119:6789/0 2024-03-29T11:22:01.804 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.823+0000 7f8354827640 10 monclient(hunting): picked mon.noname-c con 0x7f83501457f0 addr v1:172.21.15.104:6790/0 2024-03-29T11:22:01.804 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.823+0000 7f8354827640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83500ae570 con 0x7f83500a9d70 2024-03-29T11:22:01.804 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.823+0000 7f8354827640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83500ae050 con 0x7f835013f650 2024-03-29T11:22:01.804 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.824+0000 7f8354827640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83500b1610 con 0x7f83501457f0 2024-03-29T11:22:01.804 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.824+0000 7f8354827640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:01.804 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.824+0000 7f8354827640 10 monclient(hunting): authenticate will time out at 1103.075116s 2024-03-29T11:22:01.804 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.825+0000 7f834dd9b640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f835013f650 0x7f83501da670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:34592/0 (socket says 172.21.15.104:34592) 2024-03-29T11:22:01.804 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.825+0000 7f834dd9b640 1 -- 172.21.15.104:0/2969794535 learned_addr learned my addr 172.21.15.104:0/2969794535 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:22:01.804 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.825+0000 7f8346ffd640 10 client.?.objecter ms_handle_connect 0x7f835013f650 2024-03-29T11:22:01.804 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.825+0000 7f8346ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:01.804 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.825+0000 7f8346ffd640 1 -- 172.21.15.104:0/2969794535 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1625472825 0 0) 0x7f83500ae050 con 0x7f835013f650 2024-03-29T11:22:01.804 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.826+0000 7f8346ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:01.804 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.826+0000 7f8346ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:01.805 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.826+0000 7f8346ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:01.805 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.826+0000 7f8346ffd640 10 monclient(hunting): my global_id is 4309 2024-03-29T11:22:01.805 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.826+0000 7f8346ffd640 1 -- 172.21.15.104:0/2969794535 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8330004eb0 con 0x7f835013f650 2024-03-29T11:22:01.806 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.828+0000 7f8346ffd640 1 -- 172.21.15.104:0/2969794535 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 429006479 0 0) 0x7f8330004eb0 con 0x7f835013f650 2024-03-29T11:22:01.806 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.828+0000 7f8346ffd640 1 -- 172.21.15.104:0/2969794535 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f83500ae050 con 0x7f835013f650 2024-03-29T11:22:01.806 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.828+0000 7f8346ffd640 1 -- 172.21.15.104:0/2969794535 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f8340001ce0 con 0x7f835013f650 2024-03-29T11:22:01.806 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.828+0000 7f8346ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:01.806 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.828+0000 7f8346ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:01.806 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.828+0000 7f8346ffd640 10 monclient(hunting): dump: 2024-03-29T11:22:01.806 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:22:01.806 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:22:01.807 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:01.807 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:01.807 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:01.807 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:01.807 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:22:01.807 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:22:01.807 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:22:01.807 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:01.807 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.828+0000 7f8346ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:01.807 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.829+0000 7f8346ffd640 10 client.?.objecter ms_handle_connect 0x7f83500a9d70 2024-03-29T11:22:01.807 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.829+0000 7f8346ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:01.807 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.829+0000 7f8346ffd640 1 -- 172.21.15.104:0/2969794535 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2374714200 0 0) 0x7f83500ae570 con 0x7f83500a9d70 2024-03-29T11:22:01.807 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.830+0000 7f8346ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:01.807 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.830+0000 7f8346ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:01.807 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.830+0000 7f8346ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:01.808 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.830+0000 7f8346ffd640 10 monclient(hunting): my global_id is 4329 2024-03-29T11:22:01.808 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.830+0000 7f8346ffd640 1 -- 172.21.15.104:0/2969794535 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8330004eb0 con 0x7f83500a9d70 2024-03-29T11:22:01.811 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.830+0000 7f8346ffd640 1 -- 172.21.15.104:0/2969794535 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 429273852 0 0) 0x7f8330004eb0 con 0x7f83500a9d70 2024-03-29T11:22:01.811 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.830+0000 7f8346ffd640 1 -- 172.21.15.104:0/2969794535 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f83500ae570 con 0x7f83500a9d70 2024-03-29T11:22:01.811 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.830+0000 7f8346ffd640 1 -- 172.21.15.104:0/2969794535 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f8348001d00 con 0x7f83500a9d70 2024-03-29T11:22:01.811 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.830+0000 7f8346ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:01.811 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.830+0000 7f8346ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:01.811 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.830+0000 7f8346ffd640 10 monclient(hunting): dump: 2024-03-29T11:22:01.811 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:22:01.812 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:22:01.812 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:01.812 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:01.812 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:01.812 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:01.812 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:22:01.812 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:22:01.812 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:22:01.812 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:01.812 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.831+0000 7f8346ffd640 1 -- 172.21.15.104:0/2969794535 <== mon.0 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1071759498 0 0) 0x7f83500ae570 con 0x7f83500a9d70 2024-03-29T11:22:01.812 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.831+0000 7f8346ffd640 1 -- 172.21.15.104:0/2969794535 >> v1:172.21.15.104:6790/0 conn(0x7f83501457f0 legacy=0x7f83501e2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:01.812 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.831+0000 7f8346ffd640 1 -- 172.21.15.104:0/2969794535 >> v1:172.21.15.119:6789/0 conn(0x7f835013f650 legacy=0x7f83501da670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:01.812 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.831+0000 7f8346ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:22:01.812 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.831+0000 7f8346ffd640 1 monclient: found mon.a 2024-03-29T11:22:01.813 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.831+0000 7f8346ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:01.813 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.831+0000 7f8346ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:22:01.813 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.831+0000 7f8346ffd640 1 -- 172.21.15.104:0/2969794535 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83500b17a0 con 0x7f83500a9d70 2024-03-29T11:22:01.813 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.831+0000 7f8346ffd640 10 monclient: _finish_auth 0 2024-03-29T11:22:01.813 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.831+0000 7f8346ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:22:01.813 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.831+0000 7f8346ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:01.813 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.831+0000 7f8354827640 5 monclient: authenticate success, global_id 4329 2024-03-29T11:22:01.813 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.831+0000 7f8354827640 10 monclient: _renew_subs 2024-03-29T11:22:01.813 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.831+0000 7f8354827640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:22:01.813 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.831+0000 7f8354827640 1 -- 172.21.15.104:0/2969794535 --> v1:172.21.15.104:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83500ada50 con 0x7f83500a9d70 2024-03-29T11:22:01.813 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.831+0000 7f8354827640 10 client.4329.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:01.813 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.831+0000 7f8354827640 10 monclient: _renew_subs 2024-03-29T11:22:01.813 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.831+0000 7f8354827640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:22:01.813 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.831+0000 7f8354827640 1 -- 172.21.15.104:0/2969794535 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8350142270 con 0x7f83500a9d70 2024-03-29T11:22:01.813 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.831+0000 7f8354827640 1 librados: init done 2024-03-29T11:22:01.813 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.833+0000 7f8346ffd640 1 -- 172.21.15.104:0/2969794535 <== mon.0 v1:172.21.15.104:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8348004b40 con 0x7f83500a9d70 2024-03-29T11:22:01.814 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.833+0000 7f8346ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:01.814 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.832+0000 7f8354827640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:01.814 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.834+0000 7f8354827640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:01.814 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.834+0000 7f8354827640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:22:01.814 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.834+0000 7f8354827640 1 -- 172.21.15.104:0/2969794535 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83500aa650 con 0x7f83500a9d70 2024-03-29T11:22:01.814 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.834+0000 7f8346ffd640 1 -- 172.21.15.104:0/2969794535 <== mon.0 v1:172.21.15.104:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f8348004e50 con 0x7f83500a9d70 2024-03-29T11:22:01.814 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.834+0000 7f8346ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:01.814 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.834+0000 7f8346ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:01.814 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.834+0000 7f8346ffd640 10 monclient: dump: 2024-03-29T11:22:01.814 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:22:01.814 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:22:01.814 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:01.814 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:01.814 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:01.815 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:01.815 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:22:01.815 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:22:01.815 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:22:01.815 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:01.815 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.834+0000 7f8347fff640 4 set_mon_vals no callback set 2024-03-29T11:22:01.816 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.837+0000 7f8346ffd640 1 -- 172.21.15.104:0/2969794535 <== mon.0 v1:172.21.15.104:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f834801ae20 con 0x7f83500a9d70 2024-03-29T11:22:01.816 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.838+0000 7f834dd9b640 10 monclient: get_auth_request con 0x7f833006de50 auth_method 0 2024-03-29T11:22:01.816 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.838+0000 7f8346ffd640 10 client.4329.objecter ms_handle_connect 0x7f833006de50 2024-03-29T11:22:01.816 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.840+0000 7f8346ffd640 1 -- 172.21.15.104:0/2969794535 <== mon.0 v1:172.21.15.104:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f83480548c0 con 0x7f83500a9d70 2024-03-29T11:22:01.816 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.840+0000 7f8346ffd640 10 client.4329.objecter ms_dispatch 0x7f8350000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:01.816 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.840+0000 7f8346ffd640 3 client.4329.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:01.816 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.840+0000 7f8346ffd640 3 client.4329.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:01.816 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.840+0000 7f8346ffd640 20 client.4329.objecter dump_active .. 0 homeless 2024-03-29T11:22:01.824 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.847+0000 7f8346ffd640 1 -- 172.21.15.104:0/2969794535 <== mon.0 v1:172.21.15.104:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f83480503d0 con 0x7f83500a9d70 2024-03-29T11:22:01.824 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.847+0000 7f8346ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:01.824 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:01.847+0000 7f8346ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:02.149 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:02.172+0000 7f8354827640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:02.149 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:02.172+0000 7f8354827640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:02.149 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:02.172+0000 7f8354827640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:22:02.149 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:02.172+0000 7f8354827640 1 -- 172.21.15.104:0/2969794535 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f83500af6b0 con 0x7f83500a9d70 2024-03-29T11:22:02.150 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:02.174+0000 7f8346ffd640 1 -- 172.21.15.104:0/2969794535 <== mon.0 v1:172.21.15.104:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f8348052300 con 0x7f83500a9d70 2024-03-29T11:22:02.150 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:02.174+0000 7f8346ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:02.150 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:02.174+0000 7f8346ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:02.151 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:22:02.151 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:02.154 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:02.178+0000 7f8354827640 10 librados: watch_flush enter 2024-03-29T11:22:02.154 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:02.178+0000 7f8354827640 10 librados: watch_flush exit 2024-03-29T11:22:02.154 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:02.178+0000 7f8354827640 20 client.4329.objecter shutdown clearing up homeless session... 2024-03-29T11:22:02.155 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:02.178+0000 7f8354827640 10 client.4329.objecter successfully canceled tick 2024-03-29T11:22:02.155 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:02.178+0000 7f8354827640 1 -- 172.21.15.104:0/2969794535 >> v1:172.21.15.104:6816/3341126660 conn(0x7f833006de50 legacy=0x7f8330070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:02.155 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:02.178+0000 7f8354827640 10 monclient: shutdown 2024-03-29T11:22:02.155 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:02.178+0000 7f8354827640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:02.155 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:02.178+0000 7f8354827640 1 -- 172.21.15.104:0/2969794535 >> v1:172.21.15.104:6789/0 conn(0x7f83500a9d70 legacy=0x7f83500b0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:02.155 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:02.178+0000 7f8354827640 1 -- 172.21.15.104:0/2969794535 shutdown_connections 2024-03-29T11:22:02.155 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:02.178+0000 7f8354827640 1 -- 172.21.15.104:0/2969794535 >> 172.21.15.104:0/2969794535 conn(0x7f8350066a00 msgr2=0x7f835013c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:02.155 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:02.178+0000 7f8354827640 1 -- 172.21.15.104:0/2969794535 shutdown_connections 2024-03-29T11:22:02.155 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:02.178+0000 7f8354827640 1 -- 172.21.15.104:0/2969794535 wait complete. 2024-03-29T11:22:02.156 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:02.178+0000 7f8354827640 1 librados: shutdown 2024-03-29T11:22:02.171 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:08.977 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:09.124 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.148+0000 7ffb78625640 10 monclient: get_monmap_and_config 2024-03-29T11:22:09.125 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.148+0000 7ffb78625640 10 monclient: build_initial_monmap 2024-03-29T11:22:09.125 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.148+0000 7ffb78625640 10 monclient: monmap: 2024-03-29T11:22:09.125 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:22:09.125 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:22:09.125 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:22:09.149543+0000 2024-03-29T11:22:09.125 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:22:09.149543+0000 2024-03-29T11:22:09.125 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:09.125 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:09.125 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:22:09.125 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:22:09.125 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:22:09.125 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:09.127 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.151+0000 7ffb78625640 1 Processor -- start 2024-03-29T11:22:09.127 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.151+0000 7ffb78625640 1 -- start start 2024-03-29T11:22:09.127 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.151+0000 7ffb78625640 10 monclient: init 2024-03-29T11:22:09.130 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.152+0000 7ffb78625640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:09.130 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.152+0000 7ffb78625640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:22:09.130 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.152+0000 7ffb78625640 10 monclient(hunting): picked mon.noname-b con 0x7ffb700af290 addr v1:172.21.15.104:6790/0 2024-03-29T11:22:09.130 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.152+0000 7ffb78625640 10 monclient(hunting): picked mon.noname-a con 0x7ffb700ae740 addr v1:172.21.15.104:6789/0 2024-03-29T11:22:09.130 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.152+0000 7ffb78625640 10 monclient(hunting): picked mon.noname-c con 0x7ffb700af670 addr v1:172.21.15.119:6789/0 2024-03-29T11:22:09.130 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.152+0000 7ffb78625640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb700b0980 con 0x7ffb700ae740 2024-03-29T11:22:09.130 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.152+0000 7ffb78625640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb700b39f0 con 0x7ffb700af670 2024-03-29T11:22:09.130 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.152+0000 7ffb78625640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb700aebd0 con 0x7ffb700af290 2024-03-29T11:22:09.130 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.152+0000 7ffb78625640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:09.131 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.152+0000 7ffb78625640 10 monclient(hunting): authenticate will time out at 1110.403562s 2024-03-29T11:22:09.134 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.158+0000 7ffb75bc0640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7ffb700ae740 0x7ffb701439f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:39825/0 (socket says 172.21.15.104:39825) 2024-03-29T11:22:09.134 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.158+0000 7ffb75bc0640 1 -- 172.21.15.104:0/342335421 learned_addr learned my addr 172.21.15.104:0/342335421 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:22:09.138 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.161+0000 7ffb753bf640 1 -- 172.21.15.104:0/342335421 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1122341357 0 0) 0x7ffb700b0980 con 0x7ffb700ae740 2024-03-29T11:22:09.138 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.162+0000 7ffb753bf640 10 monclient(hunting): _negotiate 2024-03-29T11:22:09.138 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.162+0000 7ffb753bf640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:09.138 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.162+0000 7ffb753bf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:09.138 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.162+0000 7ffb753bf640 10 monclient(hunting): my global_id is 4332 2024-03-29T11:22:09.138 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.162+0000 7ffb753bf640 1 -- 172.21.15.104:0/342335421 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb5c003520 con 0x7ffb700ae740 2024-03-29T11:22:09.138 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.162+0000 7ffb753bf640 1 -- 172.21.15.104:0/342335421 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 999162676 0 0) 0x7ffb700b39f0 con 0x7ffb700af670 2024-03-29T11:22:09.138 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.162+0000 7ffb753bf640 10 monclient(hunting): _negotiate 2024-03-29T11:22:09.138 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.162+0000 7ffb753bf640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:09.138 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.162+0000 7ffb753bf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:09.139 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.162+0000 7ffb753bf640 10 monclient(hunting): my global_id is 4312 2024-03-29T11:22:09.139 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.162+0000 7ffb753bf640 1 -- 172.21.15.104:0/342335421 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb700b0980 con 0x7ffb700af670 2024-03-29T11:22:09.139 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.162+0000 7ffb753bf640 1 -- 172.21.15.104:0/342335421 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1760047735 0 0) 0x7ffb700aebd0 con 0x7ffb700af290 2024-03-29T11:22:09.139 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.162+0000 7ffb753bf640 10 monclient(hunting): _negotiate 2024-03-29T11:22:09.139 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.162+0000 7ffb753bf640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:09.139 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.162+0000 7ffb753bf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:09.139 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.162+0000 7ffb753bf640 10 monclient(hunting): my global_id is 4340 2024-03-29T11:22:09.139 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.162+0000 7ffb753bf640 1 -- 172.21.15.104:0/342335421 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb700b39f0 con 0x7ffb700af290 2024-03-29T11:22:09.141 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.162+0000 7ffb753bf640 1 -- 172.21.15.104:0/342335421 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 150481106 0 0) 0x7ffb5c003520 con 0x7ffb700ae740 2024-03-29T11:22:09.141 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.162+0000 7ffb753bf640 1 -- 172.21.15.104:0/342335421 >> v1:172.21.15.104:6790/0 conn(0x7ffb700af290 legacy=0x7ffb700b4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:09.141 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.162+0000 7ffb753bf640 1 -- 172.21.15.104:0/342335421 >> v1:172.21.15.119:6789/0 conn(0x7ffb700af670 legacy=0x7ffb70151e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:09.141 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.162+0000 7ffb753bf640 10 monclient: _finish_hunting 0 2024-03-29T11:22:09.141 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.162+0000 7ffb753bf640 1 monclient: found mon.noname-a 2024-03-29T11:22:09.141 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.162+0000 7ffb753bf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:09.141 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.162+0000 7ffb753bf640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.104:6789/0 2024-03-29T11:22:09.141 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.162+0000 7ffb753bf640 1 -- 172.21.15.104:0/342335421 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb700b4b40 con 0x7ffb700ae740 2024-03-29T11:22:09.141 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.162+0000 7ffb753bf640 10 monclient: _finish_auth 0 2024-03-29T11:22:09.141 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.162+0000 7ffb753bf640 10 monclient: _check_auth_tickets 2024-03-29T11:22:09.141 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.162+0000 7ffb753bf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:09.141 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.162+0000 7ffb78625640 5 monclient: authenticate success, global_id 4332 2024-03-29T11:22:09.141 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.162+0000 7ffb78625640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:09.142 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.162+0000 7ffb753bf640 1 -- 172.21.15.104:0/342335421 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7ffb600015c0 con 0x7ffb700ae740 2024-03-29T11:22:09.142 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.162+0000 7ffb753bf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:09.142 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.162+0000 7ffb753bf640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:09.142 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.162+0000 7ffb753bf640 10 monclient: dump: 2024-03-29T11:22:09.142 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:22:09.142 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:22:09.142 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:09.142 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:09.142 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:09.142 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:09.142 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:22:09.142 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:22:09.142 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:22:09.142 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:09.143 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.162+0000 7ffb78625640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:09.143 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.162+0000 7ffb753bf640 1 -- 172.21.15.104:0/342335421 <== mon.0 v1:172.21.15.104:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb60001810 con 0x7ffb700ae740 2024-03-29T11:22:09.143 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.162+0000 7ffb753bf640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:09.143 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.162+0000 7ffb78625640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:09.143 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.162+0000 7ffb78625640 4 set_mon_vals no callback set 2024-03-29T11:22:09.143 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.163+0000 7ffb753bf640 1 -- 172.21.15.104:0/342335421 <== mon.0 v1:172.21.15.104:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7ffb60001ba0 con 0x7ffb700ae740 2024-03-29T11:22:09.143 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.163+0000 7ffb753bf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:09.143 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.163+0000 7ffb753bf640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:09.143 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.163+0000 7ffb753bf640 10 monclient: dump: 2024-03-29T11:22:09.143 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:22:09.143 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:22:09.143 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:09.143 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:09.143 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:09.144 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:09.144 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:22:09.144 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:22:09.144 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:22:09.144 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:09.144 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.163+0000 7ffb78625640 10 monclient: shutdown 2024-03-29T11:22:09.144 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.163+0000 7ffb78625640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:09.144 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.163+0000 7ffb78625640 1 -- 172.21.15.104:0/342335421 >> v1:172.21.15.104:6789/0 conn(0x7ffb700ae740 legacy=0x7ffb701439f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:09.144 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.164+0000 7ffb78625640 1 -- 172.21.15.104:0/342335421 shutdown_connections 2024-03-29T11:22:09.144 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.164+0000 7ffb78625640 1 -- 172.21.15.104:0/342335421 >> 172.21.15.104:0/342335421 conn(0x7ffb70066a00 msgr2=0x7ffb70066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:09.144 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.165+0000 7ffb78625640 1 -- 172.21.15.104:0/342335421 shutdown_connections 2024-03-29T11:22:09.148 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.165+0000 7ffb78625640 1 -- 172.21.15.104:0/342335421 wait complete. 2024-03-29T11:22:09.148 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.166+0000 7ffb78625640 10 monclient: build_initial_monmap 2024-03-29T11:22:09.148 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.166+0000 7ffb78625640 10 monclient: monmap: 2024-03-29T11:22:09.148 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:22:09.148 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:09.148 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:22:09.148 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:22:09.148 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:09.148 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:09.148 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:22:09.148 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:22:09.148 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:22:09.148 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:09.148 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.166+0000 7ffb78625640 1 librados: starting msgr at 2024-03-29T11:22:09.149 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.166+0000 7ffb78625640 1 librados: starting objecter 2024-03-29T11:22:09.149 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.166+0000 7ffb78625640 1 Processor -- start 2024-03-29T11:22:09.149 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.166+0000 7ffb78625640 1 -- start start 2024-03-29T11:22:09.149 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.166+0000 7ffb78625640 1 librados: setting wanted keys 2024-03-29T11:22:09.149 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.166+0000 7ffb78625640 1 librados: calling monclient init 2024-03-29T11:22:09.149 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.166+0000 7ffb78625640 10 monclient: init 2024-03-29T11:22:09.149 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.167+0000 7ffb78625640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:09.149 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.167+0000 7ffb78625640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:22:09.149 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.167+0000 7ffb78625640 10 monclient(hunting): picked mon.noname-b con 0x7ffb700af670 addr v1:172.21.15.119:6789/0 2024-03-29T11:22:09.149 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.167+0000 7ffb78625640 10 monclient(hunting): picked mon.noname-c con 0x7ffb70070c30 addr v1:172.21.15.104:6790/0 2024-03-29T11:22:09.149 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.167+0000 7ffb78625640 10 monclient(hunting): picked mon.noname-a con 0x7ffb7013e280 addr v1:172.21.15.104:6789/0 2024-03-29T11:22:09.149 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.167+0000 7ffb78625640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb700aa380 con 0x7ffb7013e280 2024-03-29T11:22:09.149 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.167+0000 7ffb78625640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb700aa660 con 0x7ffb700af670 2024-03-29T11:22:09.149 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.167+0000 7ffb78625640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb700aa940 con 0x7ffb70070c30 2024-03-29T11:22:09.150 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.167+0000 7ffb78625640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:09.150 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.167+0000 7ffb78625640 10 monclient(hunting): authenticate will time out at 1110.418514s 2024-03-29T11:22:09.152 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.172+0000 7ffb75bc0640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7ffb70070c30 0x7ffb7006f390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:48489/0 (socket says 172.21.15.104:48489) 2024-03-29T11:22:09.152 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.172+0000 7ffb75bc0640 1 -- 172.21.15.104:0/559651754 learned_addr learned my addr 172.21.15.104:0/559651754 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:22:09.152 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.172+0000 7ffb66ffd640 10 client.?.objecter ms_handle_connect 0x7ffb70070c30 2024-03-29T11:22:09.152 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.172+0000 7ffb66ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:09.153 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.172+0000 7ffb66ffd640 1 -- 172.21.15.104:0/559651754 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3563357003 0 0) 0x7ffb700aa940 con 0x7ffb70070c30 2024-03-29T11:22:09.153 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.172+0000 7ffb66ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:09.153 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.172+0000 7ffb66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:09.153 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.172+0000 7ffb66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:09.153 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.172+0000 7ffb66ffd640 10 monclient(hunting): my global_id is 4343 2024-03-29T11:22:09.153 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.172+0000 7ffb66ffd640 1 -- 172.21.15.104:0/559651754 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb54004eb0 con 0x7ffb70070c30 2024-03-29T11:22:09.153 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.173+0000 7ffb66ffd640 1 -- 172.21.15.104:0/559651754 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4246047938 0 0) 0x7ffb54004eb0 con 0x7ffb70070c30 2024-03-29T11:22:09.153 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.173+0000 7ffb66ffd640 1 -- 172.21.15.104:0/559651754 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb700aa940 con 0x7ffb70070c30 2024-03-29T11:22:09.153 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.173+0000 7ffb66ffd640 1 -- 172.21.15.104:0/559651754 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7ffb60001580 con 0x7ffb70070c30 2024-03-29T11:22:09.153 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.173+0000 7ffb66ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:09.153 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.173+0000 7ffb66ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:09.153 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.173+0000 7ffb66ffd640 10 monclient(hunting): dump: 2024-03-29T11:22:09.153 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:22:09.153 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:22:09.153 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:09.154 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:09.154 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:09.154 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:09.154 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:22:09.154 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:22:09.154 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:22:09.154 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:09.154 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.173+0000 7ffb66ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:09.154 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.173+0000 7ffb66ffd640 10 client.?.objecter ms_handle_connect 0x7ffb7013e280 2024-03-29T11:22:09.154 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.173+0000 7ffb66ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:09.154 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.173+0000 7ffb66ffd640 1 -- 172.21.15.104:0/559651754 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2881925867 0 0) 0x7ffb700aa380 con 0x7ffb7013e280 2024-03-29T11:22:09.154 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.173+0000 7ffb66ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:09.154 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.173+0000 7ffb66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:09.154 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.173+0000 7ffb66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:09.155 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.173+0000 7ffb66ffd640 10 monclient(hunting): my global_id is 4335 2024-03-29T11:22:09.155 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.173+0000 7ffb66ffd640 1 -- 172.21.15.104:0/559651754 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb54004eb0 con 0x7ffb7013e280 2024-03-29T11:22:09.155 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.174+0000 7ffb66ffd640 1 -- 172.21.15.104:0/559651754 <== mon.2 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2968581147 0 0) 0x7ffb700aa940 con 0x7ffb70070c30 2024-03-29T11:22:09.155 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.174+0000 7ffb66ffd640 1 -- 172.21.15.104:0/559651754 >> v1:172.21.15.119:6789/0 conn(0x7ffb700af670 legacy=0x7ffb70071640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:22:09.155 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.176+0000 7ffb66ffd640 1 -- 172.21.15.104:0/559651754 >> v1:172.21.15.104:6789/0 conn(0x7ffb7013e280 legacy=0x7ffb701f77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:09.155 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.176+0000 7ffb66ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:22:09.155 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.176+0000 7ffb66ffd640 1 monclient: found mon.c 2024-03-29T11:22:09.155 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.176+0000 7ffb66ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:09.155 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.176+0000 7ffb66ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:22:09.155 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.176+0000 7ffb66ffd640 1 -- 172.21.15.104:0/559651754 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb700aad60 con 0x7ffb70070c30 2024-03-29T11:22:09.155 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.176+0000 7ffb66ffd640 10 monclient: _finish_auth 0 2024-03-29T11:22:09.155 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.176+0000 7ffb66ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:22:09.155 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.176+0000 7ffb66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:09.155 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.176+0000 7ffb78625640 5 monclient: authenticate success, global_id 4343 2024-03-29T11:22:09.155 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.176+0000 7ffb78625640 10 monclient: _renew_subs 2024-03-29T11:22:09.156 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.176+0000 7ffb78625640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:22:09.156 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.176+0000 7ffb78625640 1 -- 172.21.15.104:0/559651754 --> v1:172.21.15.104:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb7013dd70 con 0x7ffb70070c30 2024-03-29T11:22:09.156 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.176+0000 7ffb78625640 10 client.4343.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:09.156 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.176+0000 7ffb78625640 10 monclient: _renew_subs 2024-03-29T11:22:09.156 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.176+0000 7ffb78625640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:22:09.156 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.176+0000 7ffb78625640 1 -- 172.21.15.104:0/559651754 --> v1:172.21.15.104:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb701f90b0 con 0x7ffb70070c30 2024-03-29T11:22:09.156 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.176+0000 7ffb78625640 1 librados: init done 2024-03-29T11:22:09.157 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.177+0000 7ffb78625640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:09.157 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.177+0000 7ffb78625640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:09.157 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.177+0000 7ffb78625640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:22:09.157 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.177+0000 7ffb78625640 1 -- 172.21.15.104:0/559651754 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb70075f10 con 0x7ffb70070c30 2024-03-29T11:22:09.157 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.178+0000 7ffb66ffd640 1 -- 172.21.15.104:0/559651754 <== mon.2 v1:172.21.15.104:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb60004a90 con 0x7ffb70070c30 2024-03-29T11:22:09.157 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.178+0000 7ffb66ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:09.157 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.178+0000 7ffb66ffd640 1 -- 172.21.15.104:0/559651754 <== mon.2 v1:172.21.15.104:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7ffb60004ea0 con 0x7ffb70070c30 2024-03-29T11:22:09.157 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.178+0000 7ffb66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:09.157 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.178+0000 7ffb66ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:09.158 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.178+0000 7ffb66ffd640 10 monclient: dump: 2024-03-29T11:22:09.158 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:22:09.158 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:22:09.158 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:09.158 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:09.158 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:09.158 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:09.158 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:22:09.158 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:22:09.158 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:22:09.158 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:09.158 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.178+0000 7ffb67fff640 4 set_mon_vals no callback set 2024-03-29T11:22:09.158 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.179+0000 7ffb66ffd640 1 -- 172.21.15.104:0/559651754 <== mon.2 v1:172.21.15.104:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7ffb600050c0 con 0x7ffb70070c30 2024-03-29T11:22:09.159 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.180+0000 7ffb763c1640 10 monclient: get_auth_request con 0x7ffb5406db80 auth_method 0 2024-03-29T11:22:09.159 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.180+0000 7ffb66ffd640 1 -- 172.21.15.104:0/559651754 <== mon.2 v1:172.21.15.104:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7ffb60084cc0 con 0x7ffb70070c30 2024-03-29T11:22:09.159 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.180+0000 7ffb66ffd640 10 client.4343.objecter ms_dispatch 0x7ffb70000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:09.159 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.180+0000 7ffb66ffd640 3 client.4343.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:09.159 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.180+0000 7ffb66ffd640 3 client.4343.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:09.159 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.180+0000 7ffb66ffd640 20 client.4343.objecter dump_active .. 0 homeless 2024-03-29T11:22:09.159 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.180+0000 7ffb66ffd640 10 client.4343.objecter ms_handle_connect 0x7ffb5406db80 2024-03-29T11:22:09.166 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.189+0000 7ffb66ffd640 1 -- 172.21.15.104:0/559651754 <== mon.2 v1:172.21.15.104:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffb60050710 con 0x7ffb70070c30 2024-03-29T11:22:09.166 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.189+0000 7ffb66ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:09.166 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.189+0000 7ffb66ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:09.485 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.508+0000 7ffb78625640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:09.485 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.508+0000 7ffb78625640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:09.485 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.508+0000 7ffb78625640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:22:09.485 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.508+0000 7ffb78625640 1 -- 172.21.15.104:0/559651754 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb70074ed0 con 0x7ffb70070c30 2024-03-29T11:22:09.488 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.512+0000 7ffb66ffd640 1 -- 172.21.15.104:0/559651754 <== mon.2 v1:172.21.15.104:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7ffb60052640 con 0x7ffb70070c30 2024-03-29T11:22:09.488 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.512+0000 7ffb66ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:09.488 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.512+0000 7ffb66ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:09.489 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:22:09.489 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:09.492 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.515+0000 7ffb78625640 10 librados: watch_flush enter 2024-03-29T11:22:09.492 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.516+0000 7ffb78625640 10 librados: watch_flush exit 2024-03-29T11:22:09.492 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.516+0000 7ffb78625640 20 client.4343.objecter shutdown clearing up homeless session... 2024-03-29T11:22:09.492 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.516+0000 7ffb78625640 10 client.4343.objecter successfully canceled tick 2024-03-29T11:22:09.492 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.516+0000 7ffb78625640 1 -- 172.21.15.104:0/559651754 >> v1:172.21.15.104:6816/3341126660 conn(0x7ffb5406db80 legacy=0x7ffb54070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:09.493 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.516+0000 7ffb78625640 10 monclient: shutdown 2024-03-29T11:22:09.493 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.516+0000 7ffb78625640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:09.493 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.516+0000 7ffb78625640 1 -- 172.21.15.104:0/559651754 >> v1:172.21.15.104:6790/0 conn(0x7ffb70070c30 legacy=0x7ffb7006f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:09.493 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.517+0000 7ffb78625640 1 -- 172.21.15.104:0/559651754 shutdown_connections 2024-03-29T11:22:09.493 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.517+0000 7ffb78625640 1 -- 172.21.15.104:0/559651754 >> 172.21.15.104:0/559651754 conn(0x7ffb70066a00 msgr2=0x7ffb700b66a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:09.494 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.518+0000 7ffb78625640 1 -- 172.21.15.104:0/559651754 shutdown_connections 2024-03-29T11:22:09.494 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.518+0000 7ffb78625640 1 -- 172.21.15.104:0/559651754 wait complete. 2024-03-29T11:22:09.494 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:09.518+0000 7ffb78625640 1 librados: shutdown 2024-03-29T11:22:09.514 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:16.321 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:16.469 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.491+0000 7fc0871e8640 10 monclient: get_monmap_and_config 2024-03-29T11:22:16.469 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.491+0000 7fc0871e8640 10 monclient: build_initial_monmap 2024-03-29T11:22:16.469 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.491+0000 7fc0871e8640 10 monclient: monmap: 2024-03-29T11:22:16.469 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:22:16.470 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:22:16.470 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:22:16.492825+0000 2024-03-29T11:22:16.470 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:22:16.492825+0000 2024-03-29T11:22:16.470 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:16.470 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:16.470 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:22:16.470 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:22:16.470 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:22:16.470 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:16.470 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.493+0000 7fc0871e8640 1 Processor -- start 2024-03-29T11:22:16.470 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.493+0000 7fc0871e8640 1 -- start start 2024-03-29T11:22:16.470 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.493+0000 7fc0871e8640 10 monclient: init 2024-03-29T11:22:16.479 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.494+0000 7fc0871e8640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:16.479 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.494+0000 7fc0871e8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:22:16.480 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.494+0000 7fc0871e8640 10 monclient(hunting): picked mon.noname-c con 0x7fc0800a9930 addr v1:172.21.15.119:6789/0 2024-03-29T11:22:16.480 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.494+0000 7fc0871e8640 10 monclient(hunting): picked mon.noname-a con 0x7fc080149dc0 addr v1:172.21.15.104:6789/0 2024-03-29T11:22:16.480 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.494+0000 7fc0871e8640 10 monclient(hunting): picked mon.noname-b con 0x7fc08014ff40 addr v1:172.21.15.104:6790/0 2024-03-29T11:22:16.480 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.494+0000 7fc0871e8640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0800aae60 con 0x7fc080149dc0 2024-03-29T11:22:16.480 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.494+0000 7fc0871e8640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0800b4310 con 0x7fc0800a9930 2024-03-29T11:22:16.480 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.494+0000 7fc0871e8640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0800afdd0 con 0x7fc08014ff40 2024-03-29T11:22:16.480 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.494+0000 7fc0871e8640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:16.480 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.494+0000 7fc0871e8640 10 monclient(hunting): authenticate will time out at 1117.745507s 2024-03-29T11:22:16.480 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.500+0000 7fc085785640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7fc08014ff40 0x7fc080152330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:46103/0 (socket says 172.21.15.104:46103) 2024-03-29T11:22:16.480 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.500+0000 7fc085785640 1 -- 172.21.15.104:0/3867789277 learned_addr learned my addr 172.21.15.104:0/3867789277 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:22:16.480 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.501+0000 7fc07f7fe640 1 -- 172.21.15.104:0/3867789277 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2729507457 0 0) 0x7fc0800afdd0 con 0x7fc08014ff40 2024-03-29T11:22:16.480 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.501+0000 7fc07f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:16.480 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.501+0000 7fc07f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:16.480 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.501+0000 7fc07f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:16.481 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.501+0000 7fc07f7fe640 10 monclient(hunting): my global_id is 4346 2024-03-29T11:22:16.481 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.501+0000 7fc07f7fe640 1 -- 172.21.15.104:0/3867789277 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc070003520 con 0x7fc08014ff40 2024-03-29T11:22:16.481 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.501+0000 7fc07f7fe640 1 -- 172.21.15.104:0/3867789277 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1438631075 0 0) 0x7fc070003520 con 0x7fc08014ff40 2024-03-29T11:22:16.481 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.501+0000 7fc07f7fe640 1 -- 172.21.15.104:0/3867789277 >> v1:172.21.15.119:6789/0 conn(0x7fc0800a9930 legacy=0x7fc080143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:16.481 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.501+0000 7fc07f7fe640 1 -- 172.21.15.104:0/3867789277 >> v1:172.21.15.104:6789/0 conn(0x7fc080149dc0 legacy=0x7fc08014a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:16.481 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.501+0000 7fc07f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:22:16.481 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.501+0000 7fc07f7fe640 1 monclient: found mon.noname-b 2024-03-29T11:22:16.481 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.501+0000 7fc07f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:16.481 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.501+0000 7fc07f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.104:6790/0 2024-03-29T11:22:16.481 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.501+0000 7fc07f7fe640 1 -- 172.21.15.104:0/3867789277 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc08014a950 con 0x7fc08014ff40 2024-03-29T11:22:16.481 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.501+0000 7fc07f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:22:16.481 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.501+0000 7fc07f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:22:16.481 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.501+0000 7fc07f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:16.481 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.502+0000 7fc0871e8640 5 monclient: authenticate success, global_id 4346 2024-03-29T11:22:16.481 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.502+0000 7fc0871e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:16.481 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.502+0000 7fc07f7fe640 1 -- 172.21.15.104:0/3867789277 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fc078001a10 con 0x7fc08014ff40 2024-03-29T11:22:16.482 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.502+0000 7fc07f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:16.482 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.502+0000 7fc07f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:16.482 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.502+0000 7fc07f7fe640 10 monclient: dump: 2024-03-29T11:22:16.482 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:22:16.482 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:22:16.482 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:16.482 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:16.482 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:16.482 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:16.482 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:22:16.482 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:22:16.482 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:22:16.482 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:16.482 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.502+0000 7fc0871e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:16.483 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.502+0000 7fc07f7fe640 1 -- 172.21.15.104:0/3867789277 <== mon.2 v1:172.21.15.104:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc078001ce0 con 0x7fc08014ff40 2024-03-29T11:22:16.483 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.502+0000 7fc07f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:16.483 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.502+0000 7fc0871e8640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:16.483 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.502+0000 7fc0871e8640 4 set_mon_vals no callback set 2024-03-29T11:22:16.483 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.503+0000 7fc0871e8640 10 monclient: shutdown 2024-03-29T11:22:16.483 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.503+0000 7fc0871e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:16.483 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.503+0000 7fc0871e8640 1 -- 172.21.15.104:0/3867789277 >> v1:172.21.15.104:6790/0 conn(0x7fc08014ff40 legacy=0x7fc080152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:16.483 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.503+0000 7fc0871e8640 1 -- 172.21.15.104:0/3867789277 shutdown_connections 2024-03-29T11:22:16.483 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.503+0000 7fc0871e8640 1 -- 172.21.15.104:0/3867789277 >> 172.21.15.104:0/3867789277 conn(0x7fc0800b35e0 msgr2=0x7fc0800b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:16.484 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.504+0000 7fc0871e8640 1 -- 172.21.15.104:0/3867789277 shutdown_connections 2024-03-29T11:22:16.484 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.505+0000 7fc0871e8640 1 -- 172.21.15.104:0/3867789277 wait complete. 2024-03-29T11:22:16.484 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.506+0000 7fc0871e8640 10 monclient: build_initial_monmap 2024-03-29T11:22:16.484 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.506+0000 7fc0871e8640 10 monclient: monmap: 2024-03-29T11:22:16.484 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:22:16.484 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:16.484 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:22:16.484 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:22:16.484 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:16.484 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:16.484 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:22:16.484 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:22:16.485 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:22:16.485 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:16.485 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.506+0000 7fc0871e8640 1 librados: starting msgr at 2024-03-29T11:22:16.485 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.506+0000 7fc0871e8640 1 librados: starting objecter 2024-03-29T11:22:16.485 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.506+0000 7fc0871e8640 1 Processor -- start 2024-03-29T11:22:16.485 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.507+0000 7fc0871e8640 1 -- start start 2024-03-29T11:22:16.485 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.507+0000 7fc0871e8640 1 librados: setting wanted keys 2024-03-29T11:22:16.485 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.507+0000 7fc0871e8640 1 librados: calling monclient init 2024-03-29T11:22:16.485 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.507+0000 7fc0871e8640 10 monclient: init 2024-03-29T11:22:16.485 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.508+0000 7fc0871e8640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:16.485 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.508+0000 7fc0871e8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:22:16.486 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.508+0000 7fc0871e8640 10 monclient(hunting): picked mon.noname-b con 0x7fc0800a9930 addr v1:172.21.15.119:6789/0 2024-03-29T11:22:16.486 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.508+0000 7fc0871e8640 10 monclient(hunting): picked mon.noname-c con 0x7fc080149dc0 addr v1:172.21.15.104:6790/0 2024-03-29T11:22:16.486 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.508+0000 7fc0871e8640 10 monclient(hunting): picked mon.noname-a con 0x7fc08014ff40 addr v1:172.21.15.104:6789/0 2024-03-29T11:22:16.486 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.508+0000 7fc0871e8640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc080139ae0 con 0x7fc08014ff40 2024-03-29T11:22:16.486 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.508+0000 7fc0871e8640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc080139dc0 con 0x7fc0800a9930 2024-03-29T11:22:16.486 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.508+0000 7fc0871e8640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc08006ecc0 con 0x7fc080149dc0 2024-03-29T11:22:16.486 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.508+0000 7fc0871e8640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:16.486 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.508+0000 7fc0871e8640 10 monclient(hunting): authenticate will time out at 1117.759194s 2024-03-29T11:22:16.487 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.511+0000 7fc07ffff640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7fc080149dc0 0x7fc08006f8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:46225/0 (socket says 172.21.15.104:46225) 2024-03-29T11:22:16.487 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.511+0000 7fc07ffff640 1 -- 172.21.15.104:0/1161450134 learned_addr learned my addr 172.21.15.104:0/1161450134 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:22:16.488 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.512+0000 7fc07d7fa640 10 client.?.objecter ms_handle_connect 0x7fc080149dc0 2024-03-29T11:22:16.489 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.512+0000 7fc07d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:16.489 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.512+0000 7fc07d7fa640 1 -- 172.21.15.104:0/1161450134 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4002680188 0 0) 0x7fc08006ecc0 con 0x7fc080149dc0 2024-03-29T11:22:16.489 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.512+0000 7fc07d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:22:16.489 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.512+0000 7fc07d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:16.489 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.512+0000 7fc07d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:16.489 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.512+0000 7fc07d7fa640 10 monclient(hunting): my global_id is 4349 2024-03-29T11:22:16.489 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.512+0000 7fc07d7fa640 1 -- 172.21.15.104:0/1161450134 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc064004eb0 con 0x7fc080149dc0 2024-03-29T11:22:16.492 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.516+0000 7fc07d7fa640 10 client.?.objecter ms_handle_connect 0x7fc08014ff40 2024-03-29T11:22:16.492 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.516+0000 7fc07d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:16.493 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.516+0000 7fc07d7fa640 1 -- 172.21.15.104:0/1161450134 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3868155603 0 0) 0x7fc064004eb0 con 0x7fc080149dc0 2024-03-29T11:22:16.493 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.516+0000 7fc07d7fa640 1 -- 172.21.15.104:0/1161450134 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc08006ecc0 con 0x7fc080149dc0 2024-03-29T11:22:16.493 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.516+0000 7fc07d7fa640 1 -- 172.21.15.104:0/1161450134 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fc06c001a30 con 0x7fc080149dc0 2024-03-29T11:22:16.493 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.516+0000 7fc07d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:16.493 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.516+0000 7fc07d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:16.493 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.516+0000 7fc07d7fa640 10 monclient(hunting): dump: 2024-03-29T11:22:16.493 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:22:16.493 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:22:16.494 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:16.494 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:16.494 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:16.494 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:16.494 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:22:16.494 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:22:16.494 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:22:16.494 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:16.494 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.516+0000 7fc07d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:16.494 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.516+0000 7fc07d7fa640 1 -- 172.21.15.104:0/1161450134 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1493556323 0 0) 0x7fc080139ae0 con 0x7fc08014ff40 2024-03-29T11:22:16.494 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.516+0000 7fc07d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:22:16.494 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.516+0000 7fc07d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:16.494 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.516+0000 7fc07d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:16.494 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.516+0000 7fc07d7fa640 10 monclient(hunting): my global_id is 4338 2024-03-29T11:22:16.494 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.516+0000 7fc07d7fa640 1 -- 172.21.15.104:0/1161450134 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc064004eb0 con 0x7fc08014ff40 2024-03-29T11:22:16.495 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.516+0000 7fc07d7fa640 10 client.?.objecter ms_handle_connect 0x7fc0800a9930 2024-03-29T11:22:16.495 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.516+0000 7fc07d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:16.495 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.517+0000 7fc07d7fa640 1 -- 172.21.15.104:0/1161450134 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 495228113 0 0) 0x7fc064004eb0 con 0x7fc08014ff40 2024-03-29T11:22:16.495 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.517+0000 7fc07d7fa640 1 -- 172.21.15.104:0/1161450134 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc080139ae0 con 0x7fc08014ff40 2024-03-29T11:22:16.495 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.517+0000 7fc07d7fa640 1 -- 172.21.15.104:0/1161450134 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fc078004ca0 con 0x7fc08014ff40 2024-03-29T11:22:16.495 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.517+0000 7fc07d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:16.495 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.517+0000 7fc07d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:16.495 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.517+0000 7fc07d7fa640 10 monclient(hunting): dump: 2024-03-29T11:22:16.495 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:22:16.495 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:22:16.495 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:16.495 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:16.495 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:16.495 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:16.495 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:22:16.496 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:22:16.496 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:22:16.496 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:16.496 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.517+0000 7fc07d7fa640 1 -- 172.21.15.104:0/1161450134 <== mon.2 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 962375627 0 0) 0x7fc08006ecc0 con 0x7fc080149dc0 2024-03-29T11:22:16.496 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.517+0000 7fc07d7fa640 1 -- 172.21.15.104:0/1161450134 >> v1:172.21.15.119:6789/0 conn(0x7fc0800a9930 legacy=0x7fc08006f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:16.496 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.517+0000 7fc07d7fa640 1 -- 172.21.15.104:0/1161450134 >> v1:172.21.15.104:6789/0 conn(0x7fc08014ff40 legacy=0x7fc0801ee8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:16.496 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.517+0000 7fc07d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:22:16.496 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.517+0000 7fc07d7fa640 1 monclient: found mon.c 2024-03-29T11:22:16.496 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.517+0000 7fc07d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:16.496 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.517+0000 7fc07d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:22:16.496 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.517+0000 7fc07d7fa640 1 -- 172.21.15.104:0/1161450134 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0800714a0 con 0x7fc080149dc0 2024-03-29T11:22:16.496 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.517+0000 7fc07d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:22:16.496 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.517+0000 7fc07d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:22:16.496 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.517+0000 7fc07d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:16.496 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.517+0000 7fc0871e8640 5 monclient: authenticate success, global_id 4349 2024-03-29T11:22:16.497 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.517+0000 7fc0871e8640 10 monclient: _renew_subs 2024-03-29T11:22:16.497 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.517+0000 7fc0871e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:22:16.497 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.517+0000 7fc0871e8640 1 -- 172.21.15.104:0/1161450134 --> v1:172.21.15.104:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc080138df0 con 0x7fc080149dc0 2024-03-29T11:22:16.497 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.517+0000 7fc0871e8640 10 client.4349.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:16.497 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.517+0000 7fc0871e8640 10 monclient: _renew_subs 2024-03-29T11:22:16.497 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.517+0000 7fc0871e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:22:16.497 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.517+0000 7fc0871e8640 1 -- 172.21.15.104:0/1161450134 --> v1:172.21.15.104:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc080072430 con 0x7fc080149dc0 2024-03-29T11:22:16.497 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.517+0000 7fc0871e8640 1 librados: init done 2024-03-29T11:22:16.498 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.517+0000 7fc07d7fa640 1 -- 172.21.15.104:0/1161450134 <== mon.2 v1:172.21.15.104:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc06c001cf0 con 0x7fc080149dc0 2024-03-29T11:22:16.498 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.517+0000 7fc07d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:16.498 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.517+0000 7fc07d7fa640 1 -- 172.21.15.104:0/1161450134 <== mon.2 v1:172.21.15.104:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fc06c003a60 con 0x7fc080149dc0 2024-03-29T11:22:16.498 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.517+0000 7fc07d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:16.498 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.517+0000 7fc07d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:16.498 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.517+0000 7fc07d7fa640 10 monclient: dump: 2024-03-29T11:22:16.498 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:22:16.498 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:22:16.498 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:16.498 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:16.498 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:16.498 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:16.499 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:22:16.499 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:22:16.499 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:22:16.499 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:16.499 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.518+0000 7fc07e7fc640 4 set_mon_vals no callback set 2024-03-29T11:22:16.499 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.519+0000 7fc07d7fa640 1 -- 172.21.15.104:0/1161450134 <== mon.2 v1:172.21.15.104:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7fc06c003bc0 con 0x7fc080149dc0 2024-03-29T11:22:16.499 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.520+0000 7fc07d7fa640 1 -- 172.21.15.104:0/1161450134 <== mon.2 v1:172.21.15.104:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7fc06c084b50 con 0x7fc080149dc0 2024-03-29T11:22:16.499 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.520+0000 7fc07d7fa640 10 client.4349.objecter ms_dispatch 0x7fc080000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:16.499 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.520+0000 7fc07d7fa640 3 client.4349.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:16.499 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.520+0000 7fc07d7fa640 3 client.4349.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:16.499 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.520+0000 7fc07d7fa640 20 client.4349.objecter dump_active .. 0 homeless 2024-03-29T11:22:16.500 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.520+0000 7fc084f84640 10 monclient: get_auth_request con 0x7fc06406ded0 auth_method 0 2024-03-29T11:22:16.500 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.520+0000 7fc07d7fa640 10 client.4349.objecter ms_handle_connect 0x7fc06406ded0 2024-03-29T11:22:16.500 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.521+0000 7fc0871e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:16.500 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.521+0000 7fc0871e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:16.500 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.521+0000 7fc0871e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:22:16.500 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.521+0000 7fc0871e8640 1 -- 172.21.15.104:0/1161450134 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc060005080 con 0x7fc080149dc0 2024-03-29T11:22:16.510 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.529+0000 7fc07d7fa640 1 -- 172.21.15.104:0/1161450134 <== mon.2 v1:172.21.15.104:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc06c050430 con 0x7fc080149dc0 2024-03-29T11:22:16.510 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.529+0000 7fc07d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:16.510 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.529+0000 7fc07d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:16.828 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.852+0000 7fc0871e8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:16.828 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.852+0000 7fc0871e8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:16.828 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.852+0000 7fc0871e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:22:16.828 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.852+0000 7fc0871e8640 1 -- 172.21.15.104:0/1161450134 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc060005330 con 0x7fc080149dc0 2024-03-29T11:22:16.829 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.853+0000 7fc07d7fa640 1 -- 172.21.15.104:0/1161450134 <== mon.2 v1:172.21.15.104:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7fc06c052360 con 0x7fc080149dc0 2024-03-29T11:22:16.829 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.853+0000 7fc07d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:16.829 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.853+0000 7fc07d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:16.832 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:22:16.832 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:16.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.859+0000 7fc0871e8640 10 librados: watch_flush enter 2024-03-29T11:22:16.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.859+0000 7fc0871e8640 10 librados: watch_flush exit 2024-03-29T11:22:16.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.859+0000 7fc0871e8640 20 client.4349.objecter shutdown clearing up homeless session... 2024-03-29T11:22:16.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.859+0000 7fc0871e8640 10 client.4349.objecter successfully canceled tick 2024-03-29T11:22:16.836 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.859+0000 7fc0871e8640 1 -- 172.21.15.104:0/1161450134 >> v1:172.21.15.104:6816/3341126660 conn(0x7fc06406ded0 legacy=0x7fc064070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:16.836 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.859+0000 7fc0871e8640 10 monclient: shutdown 2024-03-29T11:22:16.836 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.859+0000 7fc0871e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:16.836 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.859+0000 7fc0871e8640 1 -- 172.21.15.104:0/1161450134 >> v1:172.21.15.104:6790/0 conn(0x7fc080149dc0 legacy=0x7fc08006f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:16.840 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.864+0000 7fc0871e8640 1 -- 172.21.15.104:0/1161450134 shutdown_connections 2024-03-29T11:22:16.840 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.864+0000 7fc0871e8640 1 -- 172.21.15.104:0/1161450134 >> 172.21.15.104:0/1161450134 conn(0x7fc0800b35e0 msgr2=0x7fc080140a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:16.840 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.864+0000 7fc0871e8640 1 -- 172.21.15.104:0/1161450134 shutdown_connections 2024-03-29T11:22:16.840 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.864+0000 7fc0871e8640 1 -- 172.21.15.104:0/1161450134 wait complete. 2024-03-29T11:22:16.840 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:16.864+0000 7fc0871e8640 1 librados: shutdown 2024-03-29T11:22:16.861 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:23.668 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:23.816 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.840+0000 7f17af59e640 10 monclient: get_monmap_and_config 2024-03-29T11:22:23.816 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.840+0000 7f17af59e640 10 monclient: build_initial_monmap 2024-03-29T11:22:23.816 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.840+0000 7f17af59e640 10 monclient: monmap: 2024-03-29T11:22:23.817 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:22:23.817 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:22:23.817 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:22:23.841114+0000 2024-03-29T11:22:23.817 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:22:23.841114+0000 2024-03-29T11:22:23.817 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:23.817 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:23.817 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:22:23.817 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:22:23.817 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:22:23.817 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:23.818 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.841+0000 7f17af59e640 1 Processor -- start 2024-03-29T11:22:23.818 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.841+0000 7f17af59e640 1 -- start start 2024-03-29T11:22:23.818 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.841+0000 7f17af59e640 10 monclient: init 2024-03-29T11:22:23.820 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.842+0000 7f17af59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:23.820 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.842+0000 7f17af59e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:22:23.820 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.842+0000 7f17af59e640 10 monclient(hunting): picked mon.noname-a con 0x7f17a8006a70 addr v1:172.21.15.104:6789/0 2024-03-29T11:22:23.820 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.842+0000 7f17af59e640 10 monclient(hunting): picked mon.noname-b con 0x7f17a802b100 addr v1:172.21.15.104:6790/0 2024-03-29T11:22:23.820 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.842+0000 7f17af59e640 10 monclient(hunting): picked mon.noname-c con 0x7f17a8029c30 addr v1:172.21.15.119:6789/0 2024-03-29T11:22:23.820 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.842+0000 7f17af59e640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17a802a270 con 0x7f17a8006a70 2024-03-29T11:22:23.820 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.842+0000 7f17af59e640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17a802c440 con 0x7f17a8029c30 2024-03-29T11:22:23.820 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.842+0000 7f17af59e640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17a80c9440 con 0x7f17a802b100 2024-03-29T11:22:23.820 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.842+0000 7f17af59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:23.821 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.842+0000 7f17af59e640 10 monclient(hunting): authenticate will time out at 1125.093844s 2024-03-29T11:22:23.822 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.846+0000 7f17ae59c640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f17a8006a70 0x7f17a80c8d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:47463/0 (socket says 172.21.15.104:47463) 2024-03-29T11:22:23.822 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.846+0000 7f17ae59c640 1 -- 172.21.15.104:0/761076651 learned_addr learned my addr 172.21.15.104:0/761076651 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:22:23.823 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.847+0000 7f17ad59a640 1 -- 172.21.15.104:0/761076651 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 773902407 0 0) 0x7f17a802a270 con 0x7f17a8006a70 2024-03-29T11:22:23.824 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.847+0000 7f17ad59a640 10 monclient(hunting): _negotiate 2024-03-29T11:22:23.824 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.847+0000 7f17ad59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:23.824 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.847+0000 7f17ad59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:23.824 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.847+0000 7f17ad59a640 10 monclient(hunting): my global_id is 4341 2024-03-29T11:22:23.824 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.847+0000 7f17ad59a640 1 -- 172.21.15.104:0/761076651 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1798003520 con 0x7f17a8006a70 2024-03-29T11:22:23.827 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.849+0000 7f17ad59a640 1 -- 172.21.15.104:0/761076651 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1452920657 0 0) 0x7f1798003520 con 0x7f17a8006a70 2024-03-29T11:22:23.827 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.849+0000 7f17ad59a640 1 -- 172.21.15.104:0/761076651 >> v1:172.21.15.104:6790/0 conn(0x7f17a802b100 legacy=0x7f17a80cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:23.827 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.849+0000 7f17ad59a640 1 -- 172.21.15.104:0/761076651 >> v1:172.21.15.119:6789/0 conn(0x7f17a8029c30 legacy=0x7f17a80d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:23.827 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.849+0000 7f17ad59a640 10 monclient: _finish_hunting 0 2024-03-29T11:22:23.827 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.849+0000 7f17ad59a640 1 monclient: found mon.noname-a 2024-03-29T11:22:23.827 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.849+0000 7f17ad59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:23.827 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.849+0000 7f17ad59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.104:6789/0 2024-03-29T11:22:23.827 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.849+0000 7f17ad59a640 1 -- 172.21.15.104:0/761076651 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a80c95d0 con 0x7f17a8006a70 2024-03-29T11:22:23.827 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.849+0000 7f17ad59a640 10 monclient: _finish_auth 0 2024-03-29T11:22:23.827 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.849+0000 7f17ad59a640 10 monclient: _check_auth_tickets 2024-03-29T11:22:23.827 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.849+0000 7f17ad59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:23.827 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.849+0000 7f17ad59a640 1 -- 172.21.15.104:0/761076651 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f17b004f730 con 0x7f17a8006a70 2024-03-29T11:22:23.827 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.849+0000 7f17ad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:23.828 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.849+0000 7f17ad59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:23.828 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.849+0000 7f17ad59a640 10 monclient: dump: 2024-03-29T11:22:23.828 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:22:23.828 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:22:23.828 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:23.828 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:23.828 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:23.828 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:23.828 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:22:23.828 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:22:23.829 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:22:23.829 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:23.829 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.849+0000 7f17af59e640 5 monclient: authenticate success, global_id 4341 2024-03-29T11:22:23.829 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.849+0000 7f17af59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:23.829 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.849+0000 7f17ad59a640 1 -- 172.21.15.104:0/761076651 <== mon.0 v1:172.21.15.104:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f17b00008d0 con 0x7f17a8006a70 2024-03-29T11:22:23.829 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.849+0000 7f17ad59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:23.829 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.849+0000 7f17ad59a640 1 -- 172.21.15.104:0/761076651 <== mon.0 v1:172.21.15.104:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f17b00758b0 con 0x7f17a8006a70 2024-03-29T11:22:23.829 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.849+0000 7f17ad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:23.829 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.849+0000 7f17ad59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:23.829 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.849+0000 7f17ad59a640 10 monclient: dump: 2024-03-29T11:22:23.829 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:22:23.829 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:22:23.829 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:23.830 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:23.830 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:23.830 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:23.830 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:22:23.830 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:22:23.830 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:22:23.830 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:23.830 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.849+0000 7f17af59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:23.830 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.849+0000 7f17af59e640 4 set_mon_vals no callback set 2024-03-29T11:22:23.830 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.850+0000 7f17af59e640 10 monclient: shutdown 2024-03-29T11:22:23.830 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.850+0000 7f17af59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:23.831 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.850+0000 7f17af59e640 1 -- 172.21.15.104:0/761076651 >> v1:172.21.15.104:6789/0 conn(0x7f17a8006a70 legacy=0x7f17a80c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:23.831 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.850+0000 7f17af59e640 1 -- 172.21.15.104:0/761076651 shutdown_connections 2024-03-29T11:22:23.831 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.850+0000 7f17af59e640 1 -- 172.21.15.104:0/761076651 >> 172.21.15.104:0/761076651 conn(0x7f17a80058f0 msgr2=0x7f17a8005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:23.831 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.851+0000 7f17af59e640 1 -- 172.21.15.104:0/761076651 shutdown_connections 2024-03-29T11:22:23.833 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.851+0000 7f17af59e640 1 -- 172.21.15.104:0/761076651 wait complete. 2024-03-29T11:22:23.833 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.852+0000 7f17af59e640 10 monclient: build_initial_monmap 2024-03-29T11:22:23.833 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.852+0000 7f17af59e640 10 monclient: monmap: 2024-03-29T11:22:23.833 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:22:23.833 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:23.833 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:22:23.833 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:22:23.834 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:23.834 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:23.834 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:22:23.834 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:22:23.834 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:22:23.834 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:23.834 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.852+0000 7f17af59e640 1 librados: starting msgr at 2024-03-29T11:22:23.834 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.852+0000 7f17af59e640 1 librados: starting objecter 2024-03-29T11:22:23.834 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.852+0000 7f17af59e640 1 Processor -- start 2024-03-29T11:22:23.834 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.853+0000 7f17af59e640 1 -- start start 2024-03-29T11:22:23.834 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.853+0000 7f17af59e640 1 librados: setting wanted keys 2024-03-29T11:22:23.834 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.853+0000 7f17af59e640 1 librados: calling monclient init 2024-03-29T11:22:23.834 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.853+0000 7f17af59e640 10 monclient: init 2024-03-29T11:22:23.834 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.853+0000 7f17af59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:23.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.853+0000 7f17af59e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:22:23.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.853+0000 7f17af59e640 10 monclient(hunting): picked mon.noname-a con 0x7f17a8006a70 addr v1:172.21.15.104:6789/0 2024-03-29T11:22:23.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.854+0000 7f17af59e640 10 monclient(hunting): picked mon.noname-b con 0x7f17a8029c30 addr v1:172.21.15.119:6789/0 2024-03-29T11:22:23.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.854+0000 7f17af59e640 10 monclient(hunting): picked mon.noname-c con 0x7f17a80cdbb0 addr v1:172.21.15.104:6790/0 2024-03-29T11:22:23.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.854+0000 7f17af59e640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17a80cb280 con 0x7f17a8006a70 2024-03-29T11:22:23.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.854+0000 7f17af59e640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17a80cab70 con 0x7f17a8029c30 2024-03-29T11:22:23.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.854+0000 7f17af59e640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17a80cc5e0 con 0x7f17a80cdbb0 2024-03-29T11:22:23.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.854+0000 7f17af59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:23.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.854+0000 7f17af59e640 10 monclient(hunting): authenticate will time out at 1125.105050s 2024-03-29T11:22:23.836 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.858+0000 7f17add9b640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f17a8029c30 0x7f17a80cd4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:42601/0 (socket says 172.21.15.104:42601) 2024-03-29T11:22:23.836 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.858+0000 7f17add9b640 1 -- 172.21.15.104:0/1776303763 learned_addr learned my addr 172.21.15.104:0/1776303763 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:22:23.836 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.858+0000 7f179effd640 10 client.?.objecter ms_handle_connect 0x7f17a8029c30 2024-03-29T11:22:23.836 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.858+0000 7f179effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:23.844 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.861+0000 7f179effd640 10 client.?.objecter ms_handle_connect 0x7f17a8006a70 2024-03-29T11:22:23.844 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.861+0000 7f179effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:23.844 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.861+0000 7f179effd640 1 -- 172.21.15.104:0/1776303763 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 715783607 0 0) 0x7f17a80cb280 con 0x7f17a8006a70 2024-03-29T11:22:23.844 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.861+0000 7f179effd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:23.844 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.861+0000 7f179effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:23.844 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.861+0000 7f179effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:23.844 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.861+0000 7f179effd640 10 monclient(hunting): my global_id is 4344 2024-03-29T11:22:23.844 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.861+0000 7f179effd640 1 -- 172.21.15.104:0/1776303763 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1790004eb0 con 0x7f17a8006a70 2024-03-29T11:22:23.844 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.862+0000 7f179effd640 10 client.?.objecter ms_handle_connect 0x7f17a80cdbb0 2024-03-29T11:22:23.844 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.862+0000 7f179effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:23.845 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.862+0000 7f179effd640 1 -- 172.21.15.104:0/1776303763 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2922515185 0 0) 0x7f17a80cab70 con 0x7f17a8029c30 2024-03-29T11:22:23.845 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.862+0000 7f179effd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:23.845 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.862+0000 7f179effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:23.845 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.862+0000 7f179effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:23.845 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.862+0000 7f179effd640 10 monclient(hunting): my global_id is 4318 2024-03-29T11:22:23.845 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.862+0000 7f179effd640 1 -- 172.21.15.104:0/1776303763 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17a80cb280 con 0x7f17a8029c30 2024-03-29T11:22:23.845 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.862+0000 7f179effd640 1 -- 172.21.15.104:0/1776303763 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2196898377 0 0) 0x7f17a80cc5e0 con 0x7f17a80cdbb0 2024-03-29T11:22:23.845 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.862+0000 7f179effd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:23.845 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.862+0000 7f179effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:23.845 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.862+0000 7f179effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:23.845 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.862+0000 7f179effd640 10 monclient(hunting): my global_id is 4352 2024-03-29T11:22:23.845 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.862+0000 7f179effd640 1 -- 172.21.15.104:0/1776303763 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17a80cab70 con 0x7f17a80cdbb0 2024-03-29T11:22:23.845 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.862+0000 7f179effd640 1 -- 172.21.15.104:0/1776303763 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2307757459 0 0) 0x7f1790004eb0 con 0x7f17a8006a70 2024-03-29T11:22:23.845 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.862+0000 7f179effd640 1 -- 172.21.15.104:0/1776303763 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17a80cc5e0 con 0x7f17a8006a70 2024-03-29T11:22:23.845 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.862+0000 7f179effd640 1 -- 172.21.15.104:0/1776303763 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f17b004f400 con 0x7f17a8006a70 2024-03-29T11:22:23.846 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.862+0000 7f179effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:23.846 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.862+0000 7f179effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:23.846 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.862+0000 7f179effd640 10 monclient(hunting): dump: 2024-03-29T11:22:23.846 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:22:23.846 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:22:23.846 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:23.846 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:23.846 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:23.846 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:23.846 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:22:23.846 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:22:23.846 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:22:23.846 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:23.846 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.862+0000 7f179effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:23.846 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.862+0000 7f179effd640 1 -- 172.21.15.104:0/1776303763 <== mon.0 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2996379189 0 0) 0x7f17a80cc5e0 con 0x7f17a8006a70 2024-03-29T11:22:23.847 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.862+0000 7f179effd640 1 -- 172.21.15.104:0/1776303763 >> v1:172.21.15.104:6790/0 conn(0x7f17a80cdbb0 legacy=0x7f17a816b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:23.847 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.862+0000 7f179effd640 1 -- 172.21.15.104:0/1776303763 >> v1:172.21.15.119:6789/0 conn(0x7f17a8029c30 legacy=0x7f17a80cd4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:23.847 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.862+0000 7f179effd640 10 monclient: _finish_hunting 0 2024-03-29T11:22:23.847 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.862+0000 7f179effd640 1 monclient: found mon.a 2024-03-29T11:22:23.847 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.862+0000 7f179effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:23.847 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.862+0000 7f179effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:22:23.847 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.862+0000 7f179effd640 1 -- 172.21.15.104:0/1776303763 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a80cc8c0 con 0x7f17a8006a70 2024-03-29T11:22:23.847 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.862+0000 7f179effd640 10 monclient: _finish_auth 0 2024-03-29T11:22:23.847 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.862+0000 7f179effd640 10 monclient: _check_auth_tickets 2024-03-29T11:22:23.847 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.862+0000 7f179effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:23.847 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.862+0000 7f17af59e640 5 monclient: authenticate success, global_id 4344 2024-03-29T11:22:23.847 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.862+0000 7f17af59e640 10 monclient: _renew_subs 2024-03-29T11:22:23.847 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.862+0000 7f17af59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:22:23.847 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.862+0000 7f17af59e640 1 -- 172.21.15.104:0/1776303763 --> v1:172.21.15.104:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17a80cae30 con 0x7f17a8006a70 2024-03-29T11:22:23.847 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.862+0000 7f17af59e640 10 client.4344.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:23.848 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.862+0000 7f17af59e640 10 monclient: _renew_subs 2024-03-29T11:22:23.848 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.862+0000 7f17af59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:22:23.848 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.862+0000 7f17af59e640 1 -- 172.21.15.104:0/1776303763 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f17a816cd40 con 0x7f17a8006a70 2024-03-29T11:22:23.848 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.862+0000 7f17af59e640 1 librados: init done 2024-03-29T11:22:23.848 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.864+0000 7f17af59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:23.848 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.864+0000 7f17af59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:23.848 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.864+0000 7f17af59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:22:23.848 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.864+0000 7f17af59e640 1 -- 172.21.15.104:0/1776303763 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17a80cc440 con 0x7f17a8006a70 2024-03-29T11:22:23.848 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.864+0000 7f179effd640 1 -- 172.21.15.104:0/1776303763 <== mon.0 v1:172.21.15.104:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f17b004fdd0 con 0x7f17a8006a70 2024-03-29T11:22:23.848 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.864+0000 7f179effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:23.848 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.864+0000 7f179effd640 1 -- 172.21.15.104:0/1776303763 <== mon.0 v1:172.21.15.104:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f17b0067bf0 con 0x7f17a8006a70 2024-03-29T11:22:23.848 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.864+0000 7f179effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:23.848 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.864+0000 7f179effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:23.848 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.864+0000 7f179effd640 10 monclient: dump: 2024-03-29T11:22:23.848 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:22:23.849 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:22:23.849 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:23.849 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:23.849 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:23.849 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:23.849 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:22:23.849 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:22:23.849 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:22:23.849 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:23.849 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.864+0000 7f17acd99640 4 set_mon_vals no callback set 2024-03-29T11:22:23.849 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.865+0000 7f179effd640 1 -- 172.21.15.104:0/1776303763 <== mon.0 v1:172.21.15.104:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f17b0067e10 con 0x7f17a8006a70 2024-03-29T11:22:23.850 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.866+0000 7f179effd640 1 -- 172.21.15.104:0/1776303763 <== mon.0 v1:172.21.15.104:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f17b00ff470 con 0x7f17a8006a70 2024-03-29T11:22:23.850 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.866+0000 7f179effd640 10 client.4344.objecter ms_dispatch 0x7f17b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:23.850 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.866+0000 7f179effd640 3 client.4344.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:23.850 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.866+0000 7f179effd640 3 client.4344.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:23.850 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.866+0000 7f179effd640 20 client.4344.objecter dump_active .. 0 homeless 2024-03-29T11:22:23.850 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.866+0000 7f17add9b640 10 monclient: get_auth_request con 0x7f179006d9e0 auth_method 0 2024-03-29T11:22:23.850 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.867+0000 7f179effd640 10 client.4344.objecter ms_handle_connect 0x7f179006d9e0 2024-03-29T11:22:23.853 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.876+0000 7f179effd640 1 -- 172.21.15.104:0/1776303763 <== mon.0 v1:172.21.15.104:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f17a80cc440 con 0x7f17a8006a70 2024-03-29T11:22:23.853 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.876+0000 7f179effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:23.853 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:23.876+0000 7f179effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:24.185 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:24.208+0000 7f17af59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:24.185 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:24.208+0000 7f17af59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:24.185 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:24.208+0000 7f17af59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:22:24.185 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:24.208+0000 7f17af59e640 1 -- 172.21.15.104:0/1776303763 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f17a80cc440 con 0x7f17a8006a70 2024-03-29T11:22:24.185 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:24.209+0000 7f179effd640 1 -- 172.21.15.104:0/1776303763 <== mon.0 v1:172.21.15.104:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f17a80cc440 con 0x7f17a8006a70 2024-03-29T11:22:24.185 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:24.209+0000 7f179effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:24.185 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:24.209+0000 7f179effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:24.187 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:22:24.187 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:24.193 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:24.216+0000 7f17af59e640 10 librados: watch_flush enter 2024-03-29T11:22:24.193 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:24.216+0000 7f17af59e640 10 librados: watch_flush exit 2024-03-29T11:22:24.193 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:24.216+0000 7f17af59e640 20 client.4344.objecter shutdown clearing up homeless session... 2024-03-29T11:22:24.193 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:24.216+0000 7f17af59e640 10 client.4344.objecter successfully canceled tick 2024-03-29T11:22:24.193 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:24.216+0000 7f17af59e640 1 -- 172.21.15.104:0/1776303763 >> v1:172.21.15.104:6816/3341126660 conn(0x7f179006d9e0 legacy=0x7f179006fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:24.194 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:24.216+0000 7f17af59e640 10 monclient: shutdown 2024-03-29T11:22:24.194 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:24.216+0000 7f17af59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:24.194 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:24.216+0000 7f17af59e640 1 -- 172.21.15.104:0/1776303763 >> v1:172.21.15.104:6789/0 conn(0x7f17a8006a70 legacy=0x7f17a80ca460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:24.194 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:24.217+0000 7f17af59e640 1 -- 172.21.15.104:0/1776303763 shutdown_connections 2024-03-29T11:22:24.194 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:24.217+0000 7f17af59e640 1 -- 172.21.15.104:0/1776303763 >> 172.21.15.104:0/1776303763 conn(0x7f17a80058f0 msgr2=0x7f17a80c7150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:24.194 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:24.217+0000 7f17af59e640 1 -- 172.21.15.104:0/1776303763 shutdown_connections 2024-03-29T11:22:24.194 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:24.217+0000 7f17af59e640 1 -- 172.21.15.104:0/1776303763 wait complete. 2024-03-29T11:22:24.194 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:24.217+0000 7f17af59e640 1 librados: shutdown 2024-03-29T11:22:24.212 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:31.019 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:31.175 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.197+0000 7f024c9ec640 10 monclient: get_monmap_and_config 2024-03-29T11:22:31.175 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.197+0000 7f024c9ec640 10 monclient: build_initial_monmap 2024-03-29T11:22:31.175 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.197+0000 7f024c9ec640 10 monclient: monmap: 2024-03-29T11:22:31.175 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:22:31.175 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:22:31.175 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:22:31.199712+0000 2024-03-29T11:22:31.175 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:22:31.199712+0000 2024-03-29T11:22:31.175 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:31.175 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:31.175 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:22:31.175 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:22:31.176 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:22:31.176 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:31.176 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.199+0000 7f024c9ec640 1 Processor -- start 2024-03-29T11:22:31.179 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.202+0000 7f024c9ec640 1 -- start start 2024-03-29T11:22:31.179 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.202+0000 7f024c9ec640 10 monclient: init 2024-03-29T11:22:31.180 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.203+0000 7f024c9ec640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:31.180 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.203+0000 7f024c9ec640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:22:31.180 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.203+0000 7f024c9ec640 10 monclient(hunting): picked mon.noname-c con 0x7f02400389e0 addr v1:172.21.15.119:6789/0 2024-03-29T11:22:31.180 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.203+0000 7f024c9ec640 10 monclient(hunting): picked mon.noname-b con 0x7f0240038dc0 addr v1:172.21.15.104:6790/0 2024-03-29T11:22:31.180 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.203+0000 7f024c9ec640 10 monclient(hunting): picked mon.noname-a con 0x7f0240030660 addr v1:172.21.15.104:6789/0 2024-03-29T11:22:31.180 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.203+0000 7f024c9ec640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02400c54c0 con 0x7f0240030660 2024-03-29T11:22:31.180 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.203+0000 7f024c9ec640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0240036430 con 0x7f02400389e0 2024-03-29T11:22:31.180 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.203+0000 7f024c9ec640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0240039840 con 0x7f0240038dc0 2024-03-29T11:22:31.180 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.203+0000 7f024c9ec640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:31.180 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.203+0000 7f024c9ec640 10 monclient(hunting): authenticate will time out at 1132.455253s 2024-03-29T11:22:31.184 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.207+0000 7f0246ffd640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f0240038dc0 0x7f02400cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:44023/0 (socket says 172.21.15.104:44023) 2024-03-29T11:22:31.184 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.207+0000 7f02477fe640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f02400389e0 0x7f0240033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:38679/0 (socket says 172.21.15.104:38679) 2024-03-29T11:22:31.184 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.207+0000 7f02477fe640 1 -- 172.21.15.104:0/736732684 learned_addr learned my addr 172.21.15.104:0/736732684 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:22:31.185 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.207+0000 7f02467fc640 1 -- 172.21.15.104:0/736732684 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2835244072 0 0) 0x7f0240039840 con 0x7f0240038dc0 2024-03-29T11:22:31.185 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.207+0000 7f02467fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:31.185 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.207+0000 7f02467fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:31.185 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.207+0000 7f02467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:31.185 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.207+0000 7f02467fc640 10 monclient(hunting): my global_id is 4355 2024-03-29T11:22:31.185 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.207+0000 7f02467fc640 1 -- 172.21.15.104:0/736732684 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0238003520 con 0x7f0240038dc0 2024-03-29T11:22:31.185 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.208+0000 7f02467fc640 1 -- 172.21.15.104:0/736732684 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 839258720 0 0) 0x7f0238003520 con 0x7f0240038dc0 2024-03-29T11:22:31.185 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.208+0000 7f02467fc640 1 -- 172.21.15.104:0/736732684 >> v1:172.21.15.119:6789/0 conn(0x7f02400389e0 legacy=0x7f0240033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:31.185 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.208+0000 7f02467fc640 1 -- 172.21.15.104:0/736732684 >> v1:172.21.15.104:6789/0 conn(0x7f0240030660 legacy=0x7f02400d3170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:31.188 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.210+0000 7f02467fc640 10 monclient: _finish_hunting 0 2024-03-29T11:22:31.188 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.210+0000 7f02467fc640 1 monclient: found mon.noname-b 2024-03-29T11:22:31.188 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.210+0000 7f02467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:31.188 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.210+0000 7f02467fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.104:6790/0 2024-03-29T11:22:31.189 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.210+0000 7f02467fc640 1 -- 172.21.15.104:0/736732684 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0240035690 con 0x7f0240038dc0 2024-03-29T11:22:31.189 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.210+0000 7f02467fc640 10 monclient: _finish_auth 0 2024-03-29T11:22:31.189 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.210+0000 7f02467fc640 10 monclient: _check_auth_tickets 2024-03-29T11:22:31.189 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.210+0000 7f02467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:31.189 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.210+0000 7f024c9ec640 5 monclient: authenticate success, global_id 4355 2024-03-29T11:22:31.189 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.210+0000 7f024c9ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:31.189 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.210+0000 7f02467fc640 1 -- 172.21.15.104:0/736732684 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f0234001690 con 0x7f0240038dc0 2024-03-29T11:22:31.189 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.210+0000 7f02467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:31.189 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.210+0000 7f02467fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:31.189 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.210+0000 7f02467fc640 10 monclient: dump: 2024-03-29T11:22:31.189 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:22:31.189 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:22:31.189 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:31.189 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:31.189 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:31.190 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:31.190 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:22:31.190 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:22:31.190 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:22:31.190 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:31.190 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.211+0000 7f024c9ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:31.191 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.212+0000 7f02467fc640 1 -- 172.21.15.104:0/736732684 <== mon.2 v1:172.21.15.104:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f02340018f0 con 0x7f0240038dc0 2024-03-29T11:22:31.191 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.212+0000 7f02467fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:31.191 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.212+0000 7f024c9ec640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:31.191 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.212+0000 7f024c9ec640 4 set_mon_vals no callback set 2024-03-29T11:22:31.191 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.213+0000 7f024c9ec640 10 monclient: shutdown 2024-03-29T11:22:31.191 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.213+0000 7f024c9ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:31.191 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.213+0000 7f024c9ec640 1 -- 172.21.15.104:0/736732684 >> v1:172.21.15.104:6790/0 conn(0x7f0240038dc0 legacy=0x7f02400cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:31.191 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.214+0000 7f024c9ec640 1 -- 172.21.15.104:0/736732684 shutdown_connections 2024-03-29T11:22:31.191 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.214+0000 7f024c9ec640 1 -- 172.21.15.104:0/736732684 >> 172.21.15.104:0/736732684 conn(0x7f0240005c00 msgr2=0x7f0240006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:31.192 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.214+0000 7f024c9ec640 1 -- 172.21.15.104:0/736732684 shutdown_connections 2024-03-29T11:22:31.192 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.215+0000 7f024c9ec640 1 -- 172.21.15.104:0/736732684 wait complete. 2024-03-29T11:22:31.193 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.216+0000 7f024c9ec640 10 monclient: build_initial_monmap 2024-03-29T11:22:31.193 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.216+0000 7f024c9ec640 10 monclient: monmap: 2024-03-29T11:22:31.194 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:22:31.194 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:31.194 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:22:31.194 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:22:31.194 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:31.194 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:31.194 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:22:31.194 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:22:31.194 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:22:31.194 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:31.194 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.216+0000 7f024c9ec640 1 librados: starting msgr at 2024-03-29T11:22:31.194 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.216+0000 7f024c9ec640 1 librados: starting objecter 2024-03-29T11:22:31.194 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.216+0000 7f024c9ec640 1 Processor -- start 2024-03-29T11:22:31.194 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.217+0000 7f024c9ec640 1 -- start start 2024-03-29T11:22:31.194 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.217+0000 7f024c9ec640 1 librados: setting wanted keys 2024-03-29T11:22:31.195 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.217+0000 7f024c9ec640 1 librados: calling monclient init 2024-03-29T11:22:31.195 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.217+0000 7f024c9ec640 10 monclient: init 2024-03-29T11:22:31.195 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.217+0000 7f024c9ec640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:31.195 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.217+0000 7f024c9ec640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:22:31.195 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.217+0000 7f024c9ec640 10 monclient(hunting): picked mon.noname-a con 0x7f0240030660 addr v1:172.21.15.104:6789/0 2024-03-29T11:22:31.195 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.217+0000 7f024c9ec640 10 monclient(hunting): picked mon.noname-c con 0x7f02400389e0 addr v1:172.21.15.104:6790/0 2024-03-29T11:22:31.195 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.217+0000 7f024c9ec640 10 monclient(hunting): picked mon.noname-b con 0x7f02400c7e60 addr v1:172.21.15.119:6789/0 2024-03-29T11:22:31.195 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.217+0000 7f024c9ec640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02400c7170 con 0x7f0240030660 2024-03-29T11:22:31.195 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.217+0000 7f024c9ec640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02400c3c90 con 0x7f02400c7e60 2024-03-29T11:22:31.195 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.217+0000 7f024c9ec640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02400c6c90 con 0x7f02400389e0 2024-03-29T11:22:31.196 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.217+0000 7f024c9ec640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:31.196 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.217+0000 7f024c9ec640 10 monclient(hunting): authenticate will time out at 1132.469949s 2024-03-29T11:22:31.201 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.223+0000 7f02477fe640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f0240030660 0x7f02400c7600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:42077/0 (socket says 172.21.15.104:42077) 2024-03-29T11:22:31.201 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.223+0000 7f02477fe640 1 -- 172.21.15.104:0/2668382468 learned_addr learned my addr 172.21.15.104:0/2668382468 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:22:31.201 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.224+0000 7f0233fff640 10 client.?.objecter ms_handle_connect 0x7f0240030660 2024-03-29T11:22:31.201 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.224+0000 7f0233fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:31.204 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.224+0000 7f0233fff640 1 -- 172.21.15.104:0/2668382468 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 879686699 0 0) 0x7f02400c7170 con 0x7f0240030660 2024-03-29T11:22:31.204 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.224+0000 7f0233fff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:31.204 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.224+0000 7f0233fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:31.204 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.224+0000 7f0233fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:31.204 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.224+0000 7f0233fff640 10 monclient(hunting): my global_id is 4347 2024-03-29T11:22:31.204 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.224+0000 7f0233fff640 1 -- 172.21.15.104:0/2668382468 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f022c004eb0 con 0x7f0240030660 2024-03-29T11:22:31.205 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.225+0000 7f0233fff640 1 -- 172.21.15.104:0/2668382468 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1547425449 0 0) 0x7f022c004eb0 con 0x7f0240030660 2024-03-29T11:22:31.205 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.225+0000 7f0233fff640 1 -- 172.21.15.104:0/2668382468 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02400c7170 con 0x7f0240030660 2024-03-29T11:22:31.205 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.225+0000 7f0233fff640 1 -- 172.21.15.104:0/2668382468 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f024804fdd0 con 0x7f0240030660 2024-03-29T11:22:31.205 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.225+0000 7f0233fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:31.205 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.225+0000 7f0233fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:31.205 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.225+0000 7f0233fff640 10 monclient(hunting): dump: 2024-03-29T11:22:31.205 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:22:31.205 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:22:31.205 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:31.205 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:31.205 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:31.205 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:31.205 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:22:31.205 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:22:31.205 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:22:31.205 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:31.206 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.225+0000 7f0233fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:31.206 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.225+0000 7f0233fff640 10 client.?.objecter ms_handle_connect 0x7f02400c7e60 2024-03-29T11:22:31.206 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.225+0000 7f0233fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:31.206 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.225+0000 7f0233fff640 1 -- 172.21.15.104:0/2668382468 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 334198776 0 0) 0x7f02400c3c90 con 0x7f02400c7e60 2024-03-29T11:22:31.206 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.225+0000 7f0233fff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:31.206 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.225+0000 7f0233fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:31.206 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.225+0000 7f0233fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:31.206 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.225+0000 7f0233fff640 10 monclient(hunting): my global_id is 4321 2024-03-29T11:22:31.206 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.225+0000 7f0233fff640 1 -- 172.21.15.104:0/2668382468 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f022c004eb0 con 0x7f02400c7e60 2024-03-29T11:22:31.206 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.226+0000 7f0233fff640 1 -- 172.21.15.104:0/2668382468 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2257771087 0 0) 0x7f022c004eb0 con 0x7f02400c7e60 2024-03-29T11:22:31.206 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.226+0000 7f0233fff640 1 -- 172.21.15.104:0/2668382468 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02400c3c90 con 0x7f02400c7e60 2024-03-29T11:22:31.206 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.226+0000 7f0233fff640 1 -- 172.21.15.104:0/2668382468 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f023c0017b0 con 0x7f02400c7e60 2024-03-29T11:22:31.206 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.226+0000 7f0233fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:31.207 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.226+0000 7f0233fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:31.207 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.226+0000 7f0233fff640 10 monclient(hunting): dump: 2024-03-29T11:22:31.207 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:22:31.207 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:22:31.207 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:31.207 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:31.207 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:31.207 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:31.207 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:22:31.207 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:22:31.207 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:22:31.207 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:31.207 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.226+0000 7f0233fff640 1 -- 172.21.15.104:0/2668382468 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2994649596 0 0) 0x7f02400c3c90 con 0x7f02400c7e60 2024-03-29T11:22:31.207 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.226+0000 7f0233fff640 1 -- 172.21.15.104:0/2668382468 >> v1:172.21.15.104:6790/0 conn(0x7f02400389e0 legacy=0x7f02400c2e70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:22:31.210 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.228+0000 7f0233fff640 1 -- 172.21.15.104:0/2668382468 >> v1:172.21.15.104:6789/0 conn(0x7f0240030660 legacy=0x7f02400c7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:31.210 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.228+0000 7f0233fff640 10 monclient: _finish_hunting 0 2024-03-29T11:22:31.211 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.228+0000 7f0233fff640 1 monclient: found mon.b 2024-03-29T11:22:31.211 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.228+0000 7f0233fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:31.211 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.228+0000 7f0233fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:22:31.211 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.228+0000 7f0233fff640 1 -- 172.21.15.104:0/2668382468 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02400c6e20 con 0x7f02400c7e60 2024-03-29T11:22:31.211 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.228+0000 7f0233fff640 10 monclient: _finish_auth 0 2024-03-29T11:22:31.211 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.228+0000 7f0233fff640 10 monclient: _check_auth_tickets 2024-03-29T11:22:31.211 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.228+0000 7f0233fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:31.211 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.228+0000 7f0233fff640 1 -- 172.21.15.104:0/2668382468 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f023c001cc0 con 0x7f02400c7e60 2024-03-29T11:22:31.211 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.228+0000 7f0233fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:31.211 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.228+0000 7f0233fff640 1 -- 172.21.15.104:0/2668382468 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f023c003790 con 0x7f02400c7e60 2024-03-29T11:22:31.211 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.228+0000 7f0233fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:31.211 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.228+0000 7f0233fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:31.211 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.228+0000 7f0233fff640 10 monclient: dump: 2024-03-29T11:22:31.211 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:22:31.211 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:22:31.212 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:31.212 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:31.212 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:31.212 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:31.212 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:22:31.212 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:22:31.212 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:22:31.212 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:31.212 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.228+0000 7f024c9ec640 5 monclient: authenticate success, global_id 4321 2024-03-29T11:22:31.212 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.228+0000 7f024c9ec640 10 monclient: _renew_subs 2024-03-29T11:22:31.212 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.228+0000 7f024c9ec640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:22:31.212 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.228+0000 7f024c9ec640 1 -- 172.21.15.104:0/2668382468 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02400c5650 con 0x7f02400c7e60 2024-03-29T11:22:31.212 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.228+0000 7f024c9ec640 10 client.4321.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:31.212 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.228+0000 7f024c9ec640 10 monclient: _renew_subs 2024-03-29T11:22:31.213 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.228+0000 7f024c9ec640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:22:31.213 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.228+0000 7f024c9ec640 1 -- 172.21.15.104:0/2668382468 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f02400ca740 con 0x7f02400c7e60 2024-03-29T11:22:31.213 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.228+0000 7f024c9ec640 1 librados: init done 2024-03-29T11:22:31.213 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.231+0000 7f0245ffb640 4 set_mon_vals no callback set 2024-03-29T11:22:31.213 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.232+0000 7f024c9ec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:31.213 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.232+0000 7f024c9ec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:31.213 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.232+0000 7f024c9ec640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:22:31.213 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.232+0000 7f024c9ec640 1 -- 172.21.15.104:0/2668382468 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0228005080 con 0x7f02400c7e60 2024-03-29T11:22:31.213 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.232+0000 7f0233fff640 1 -- 172.21.15.104:0/2668382468 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f023c019760 con 0x7f02400c7e60 2024-03-29T11:22:31.213 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.233+0000 7f0233fff640 1 -- 172.21.15.104:0/2668382468 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f023c083690 con 0x7f02400c7e60 2024-03-29T11:22:31.213 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.233+0000 7f0233fff640 10 client.4321.objecter ms_dispatch 0x7f0248000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:31.213 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.233+0000 7f0233fff640 3 client.4321.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:31.213 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.233+0000 7f0233fff640 3 client.4321.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:31.213 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.233+0000 7f0233fff640 20 client.4321.objecter dump_active .. 0 homeless 2024-03-29T11:22:31.213 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.233+0000 7f02477fe640 10 monclient: get_auth_request con 0x7f022c06de00 auth_method 0 2024-03-29T11:22:31.213 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.233+0000 7f0233fff640 10 client.4321.objecter ms_handle_connect 0x7f022c06de00 2024-03-29T11:22:31.217 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.240+0000 7f0233fff640 1 -- 172.21.15.104:0/2668382468 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f023c04eda0 con 0x7f02400c7e60 2024-03-29T11:22:31.217 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.240+0000 7f0233fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:31.217 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.240+0000 7f0233fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:31.538 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.557+0000 7f024c9ec640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:31.538 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.557+0000 7f024c9ec640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:31.538 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.557+0000 7f024c9ec640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:22:31.538 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.557+0000 7f024c9ec640 1 -- 172.21.15.104:0/2668382468 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0228005330 con 0x7f02400c7e60 2024-03-29T11:22:31.538 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.560+0000 7f0233fff640 1 -- 172.21.15.104:0/2668382468 <== mon.1 v1:172.21.15.119:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f023c050ea0 con 0x7f02400c7e60 2024-03-29T11:22:31.538 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.560+0000 7f0233fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:31.539 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.560+0000 7f0233fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:31.541 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:22:31.541 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:31.542 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.565+0000 7f024c9ec640 10 librados: watch_flush enter 2024-03-29T11:22:31.543 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.566+0000 7f024c9ec640 10 librados: watch_flush exit 2024-03-29T11:22:31.543 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.566+0000 7f024c9ec640 20 client.4321.objecter shutdown clearing up homeless session... 2024-03-29T11:22:31.543 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.566+0000 7f024c9ec640 10 client.4321.objecter successfully canceled tick 2024-03-29T11:22:31.543 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.566+0000 7f024c9ec640 1 -- 172.21.15.104:0/2668382468 >> v1:172.21.15.104:6816/3341126660 conn(0x7f022c06de00 legacy=0x7f022c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:31.543 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.566+0000 7f024c9ec640 10 monclient: shutdown 2024-03-29T11:22:31.543 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.566+0000 7f024c9ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:31.543 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.566+0000 7f024c9ec640 1 -- 172.21.15.104:0/2668382468 >> v1:172.21.15.119:6789/0 conn(0x7f02400c7e60 legacy=0x7f02400c3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:31.546 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.569+0000 7f024c9ec640 1 -- 172.21.15.104:0/2668382468 shutdown_connections 2024-03-29T11:22:31.546 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.569+0000 7f024c9ec640 1 -- 172.21.15.104:0/2668382468 >> 172.21.15.104:0/2668382468 conn(0x7f0240005c00 msgr2=0x7f02400c4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:31.547 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.570+0000 7f024c9ec640 1 -- 172.21.15.104:0/2668382468 shutdown_connections 2024-03-29T11:22:31.547 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.570+0000 7f024c9ec640 1 -- 172.21.15.104:0/2668382468 wait complete. 2024-03-29T11:22:31.548 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:31.570+0000 7f024c9ec640 1 librados: shutdown 2024-03-29T11:22:31.565 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:38.372 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:38.523 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.546+0000 7f98e0ecf640 10 monclient: get_monmap_and_config 2024-03-29T11:22:38.523 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.546+0000 7f98e0ecf640 10 monclient: build_initial_monmap 2024-03-29T11:22:38.523 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.546+0000 7f98e0ecf640 10 monclient: monmap: 2024-03-29T11:22:38.523 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:22:38.523 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:22:38.523 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:22:38.547013+0000 2024-03-29T11:22:38.524 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:22:38.547013+0000 2024-03-29T11:22:38.524 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:38.524 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:38.524 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:22:38.524 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:22:38.524 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:22:38.524 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:38.526 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.547+0000 7f98e0ecf640 1 Processor -- start 2024-03-29T11:22:38.526 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.547+0000 7f98e0ecf640 1 -- start start 2024-03-29T11:22:38.526 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.547+0000 7f98e0ecf640 10 monclient: init 2024-03-29T11:22:38.526 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.548+0000 7f98e0ecf640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:38.526 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.548+0000 7f98e0ecf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:22:38.526 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.548+0000 7f98e0ecf640 10 monclient(hunting): picked mon.noname-b con 0x7f98dc071de0 addr v1:172.21.15.104:6790/0 2024-03-29T11:22:38.526 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.548+0000 7f98e0ecf640 10 monclient(hunting): picked mon.noname-c con 0x7f98dc0721c0 addr v1:172.21.15.119:6789/0 2024-03-29T11:22:38.526 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.548+0000 7f98e0ecf640 10 monclient(hunting): picked mon.noname-a con 0x7f98dc145ae0 addr v1:172.21.15.104:6789/0 2024-03-29T11:22:38.526 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.548+0000 7f98e0ecf640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98dc145650 con 0x7f98dc145ae0 2024-03-29T11:22:38.526 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.548+0000 7f98e0ecf640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98dc0ae260 con 0x7f98dc0721c0 2024-03-29T11:22:38.526 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.548+0000 7f98e0ecf640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98dc1420d0 con 0x7f98dc071de0 2024-03-29T11:22:38.526 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.548+0000 7f98e0ecf640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:38.527 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.548+0000 7f98e0ecf640 10 monclient(hunting): authenticate will time out at 1139.799623s 2024-03-29T11:22:38.527 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.551+0000 7f98dad9d640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f98dc145ae0 0x7f98dc145f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:45042/0 (socket says 172.21.15.104:45042) 2024-03-29T11:22:38.528 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.551+0000 7f98dad9d640 1 -- 172.21.15.104:0/3475118220 learned_addr learned my addr 172.21.15.104:0/3475118220 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:22:38.530 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.552+0000 7f98d959a640 1 -- 172.21.15.104:0/3475118220 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2786788548 0 0) 0x7f98dc145650 con 0x7f98dc145ae0 2024-03-29T11:22:38.531 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.552+0000 7f98d959a640 10 monclient(hunting): _negotiate 2024-03-29T11:22:38.531 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.552+0000 7f98d959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:38.531 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.552+0000 7f98d959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:38.531 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.552+0000 7f98d959a640 10 monclient(hunting): my global_id is 4350 2024-03-29T11:22:38.531 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.552+0000 7f98d959a640 1 -- 172.21.15.104:0/3475118220 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98c4003520 con 0x7f98dc145ae0 2024-03-29T11:22:38.531 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.552+0000 7f98d959a640 1 -- 172.21.15.104:0/3475118220 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 653932413 0 0) 0x7f98dc1420d0 con 0x7f98dc071de0 2024-03-29T11:22:38.531 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.552+0000 7f98d959a640 10 monclient(hunting): _negotiate 2024-03-29T11:22:38.532 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.552+0000 7f98d959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:38.532 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.552+0000 7f98d959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:38.532 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.552+0000 7f98d959a640 10 monclient(hunting): my global_id is 4358 2024-03-29T11:22:38.532 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.552+0000 7f98d959a640 1 -- 172.21.15.104:0/3475118220 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98dc145650 con 0x7f98dc071de0 2024-03-29T11:22:38.532 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.552+0000 7f98d959a640 1 -- 172.21.15.104:0/3475118220 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 270467332 0 0) 0x7f98dc145650 con 0x7f98dc071de0 2024-03-29T11:22:38.532 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.552+0000 7f98d959a640 1 -- 172.21.15.104:0/3475118220 >> v1:172.21.15.119:6789/0 conn(0x7f98dc0721c0 legacy=0x7f98dc13f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:38.532 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.552+0000 7f98d959a640 1 -- 172.21.15.104:0/3475118220 >> v1:172.21.15.104:6789/0 conn(0x7f98dc145ae0 legacy=0x7f98dc145f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:38.533 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.552+0000 7f98d959a640 10 monclient: _finish_hunting 0 2024-03-29T11:22:38.533 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.552+0000 7f98d959a640 1 monclient: found mon.noname-b 2024-03-29T11:22:38.533 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.552+0000 7f98d959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:38.533 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.552+0000 7f98d959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.104:6790/0 2024-03-29T11:22:38.533 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.552+0000 7f98d959a640 1 -- 172.21.15.104:0/3475118220 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98dc0b0590 con 0x7f98dc071de0 2024-03-29T11:22:38.533 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.552+0000 7f98d959a640 10 monclient: _finish_auth 0 2024-03-29T11:22:38.533 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.552+0000 7f98d959a640 10 monclient: _check_auth_tickets 2024-03-29T11:22:38.533 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.552+0000 7f98d959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:38.533 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.553+0000 7f98d959a640 1 -- 172.21.15.104:0/3475118220 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f98cc001670 con 0x7f98dc071de0 2024-03-29T11:22:38.533 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.553+0000 7f98e0ecf640 5 monclient: authenticate success, global_id 4358 2024-03-29T11:22:38.533 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.553+0000 7f98d959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:38.533 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.553+0000 7f98d959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:38.533 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.553+0000 7f98d959a640 10 monclient: dump: 2024-03-29T11:22:38.533 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:22:38.533 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:22:38.534 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:38.534 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:38.534 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:38.534 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:38.534 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:22:38.534 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:22:38.534 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:22:38.534 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:38.534 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.553+0000 7f98e0ecf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:38.534 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.553+0000 7f98d959a640 1 -- 172.21.15.104:0/3475118220 <== mon.2 v1:172.21.15.104:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98cc001850 con 0x7f98dc071de0 2024-03-29T11:22:38.534 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.553+0000 7f98d959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:38.534 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.553+0000 7f98e0ecf640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:38.535 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.553+0000 7f98e0ecf640 4 set_mon_vals no callback set 2024-03-29T11:22:38.535 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.553+0000 7f98d959a640 1 -- 172.21.15.104:0/3475118220 <== mon.2 v1:172.21.15.104:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f98cc001be0 con 0x7f98dc071de0 2024-03-29T11:22:38.535 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.553+0000 7f98d959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:38.535 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.553+0000 7f98d959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:38.535 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.553+0000 7f98d959a640 10 monclient: dump: 2024-03-29T11:22:38.535 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:22:38.535 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:22:38.535 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:38.535 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:38.535 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:38.535 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:38.535 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:22:38.535 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:22:38.535 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:22:38.535 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:38.535 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.554+0000 7f98e0ecf640 10 monclient: shutdown 2024-03-29T11:22:38.535 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.554+0000 7f98e0ecf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:38.536 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.554+0000 7f98e0ecf640 1 -- 172.21.15.104:0/3475118220 >> v1:172.21.15.104:6790/0 conn(0x7f98dc071de0 legacy=0x7f98dc0aed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:38.536 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.554+0000 7f98e0ecf640 1 -- 172.21.15.104:0/3475118220 shutdown_connections 2024-03-29T11:22:38.536 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.555+0000 7f98e0ecf640 1 -- 172.21.15.104:0/3475118220 >> 172.21.15.104:0/3475118220 conn(0x7f98dc06b660 msgr2=0x7f98dc06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:38.536 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.556+0000 7f98e0ecf640 1 -- 172.21.15.104:0/3475118220 shutdown_connections 2024-03-29T11:22:38.536 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.556+0000 7f98e0ecf640 1 -- 172.21.15.104:0/3475118220 wait complete. 2024-03-29T11:22:38.536 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.557+0000 7f98e0ecf640 10 monclient: build_initial_monmap 2024-03-29T11:22:38.536 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.557+0000 7f98e0ecf640 10 monclient: monmap: 2024-03-29T11:22:38.536 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:22:38.537 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:38.537 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:22:38.537 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:22:38.537 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:38.537 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:38.537 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:22:38.537 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:22:38.537 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:22:38.537 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:38.537 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.557+0000 7f98e0ecf640 1 librados: starting msgr at 2024-03-29T11:22:38.537 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.558+0000 7f98e0ecf640 1 librados: starting objecter 2024-03-29T11:22:38.537 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.558+0000 7f98e0ecf640 1 Processor -- start 2024-03-29T11:22:38.537 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.558+0000 7f98e0ecf640 1 -- start start 2024-03-29T11:22:38.537 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.558+0000 7f98e0ecf640 1 librados: setting wanted keys 2024-03-29T11:22:38.537 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.558+0000 7f98e0ecf640 1 librados: calling monclient init 2024-03-29T11:22:38.537 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.558+0000 7f98e0ecf640 10 monclient: init 2024-03-29T11:22:38.539 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.559+0000 7f98e0ecf640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:38.539 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.559+0000 7f98e0ecf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:22:38.539 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.559+0000 7f98e0ecf640 10 monclient(hunting): picked mon.noname-c con 0x7f98dc071de0 addr v1:172.21.15.104:6790/0 2024-03-29T11:22:38.539 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.559+0000 7f98e0ecf640 10 monclient(hunting): picked mon.noname-b con 0x7f98dc145ae0 addr v1:172.21.15.119:6789/0 2024-03-29T11:22:38.539 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.559+0000 7f98e0ecf640 10 monclient(hunting): picked mon.noname-a con 0x7f98dc06de90 addr v1:172.21.15.104:6789/0 2024-03-29T11:22:38.539 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.559+0000 7f98e0ecf640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98dc06f3d0 con 0x7f98dc06de90 2024-03-29T11:22:38.539 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.559+0000 7f98e0ecf640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98dc071c10 con 0x7f98dc145ae0 2024-03-29T11:22:38.539 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.559+0000 7f98e0ecf640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98dc06f620 con 0x7f98dc071de0 2024-03-29T11:22:38.539 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.559+0000 7f98e0ecf640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:38.539 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.559+0000 7f98e0ecf640 10 monclient(hunting): authenticate will time out at 1139.810099s 2024-03-29T11:22:38.540 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.561+0000 7f98dad9d640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f98dc06de90 0x7f98dc1e5fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:51372/0 (socket says 172.21.15.104:51372) 2024-03-29T11:22:38.540 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.562+0000 7f98dad9d640 1 -- 172.21.15.104:0/3476831778 learned_addr learned my addr 172.21.15.104:0/3476831778 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:22:38.540 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.562+0000 7f98caffd640 10 client.?.objecter ms_handle_connect 0x7f98dc06de90 2024-03-29T11:22:38.540 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.562+0000 7f98caffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:38.540 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.562+0000 7f98caffd640 1 -- 172.21.15.104:0/3476831778 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1388531164 0 0) 0x7f98dc06f3d0 con 0x7f98dc06de90 2024-03-29T11:22:38.540 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.562+0000 7f98caffd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:38.540 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.562+0000 7f98caffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:38.540 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.562+0000 7f98caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:38.540 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.562+0000 7f98caffd640 10 monclient(hunting): my global_id is 4353 2024-03-29T11:22:38.540 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.562+0000 7f98caffd640 1 -- 172.21.15.104:0/3476831778 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98bc004eb0 con 0x7f98dc06de90 2024-03-29T11:22:38.541 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.563+0000 7f98caffd640 1 -- 172.21.15.104:0/3476831778 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1373452166 0 0) 0x7f98bc004eb0 con 0x7f98dc06de90 2024-03-29T11:22:38.541 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.563+0000 7f98caffd640 1 -- 172.21.15.104:0/3476831778 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98dc06f3d0 con 0x7f98dc06de90 2024-03-29T11:22:38.541 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.563+0000 7f98caffd640 1 -- 172.21.15.104:0/3476831778 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f98d4001cd0 con 0x7f98dc06de90 2024-03-29T11:22:38.541 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.563+0000 7f98caffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:38.541 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.563+0000 7f98caffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:38.541 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.563+0000 7f98caffd640 10 monclient(hunting): dump: 2024-03-29T11:22:38.541 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:22:38.541 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:22:38.541 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:38.541 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:38.541 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:38.542 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:38.542 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:22:38.542 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:22:38.542 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:22:38.542 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:38.542 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.563+0000 7f98caffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:38.542 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.564+0000 7f98caffd640 1 -- 172.21.15.104:0/3476831778 <== mon.0 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2638217197 0 0) 0x7f98dc06f3d0 con 0x7f98dc06de90 2024-03-29T11:22:38.542 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.564+0000 7f98caffd640 1 -- 172.21.15.104:0/3476831778 >> v1:172.21.15.104:6790/0 conn(0x7f98dc071de0 legacy=0x7f98dc06fa40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:22:38.543 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.564+0000 7f98caffd640 1 -- 172.21.15.104:0/3476831778 >> v1:172.21.15.119:6789/0 conn(0x7f98dc145ae0 legacy=0x7f98dc138d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:38.543 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.564+0000 7f98caffd640 10 monclient: _finish_hunting 0 2024-03-29T11:22:38.543 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.564+0000 7f98caffd640 1 monclient: found mon.a 2024-03-29T11:22:38.543 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.564+0000 7f98caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:38.543 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.564+0000 7f98caffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:22:38.543 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.564+0000 7f98caffd640 1 -- 172.21.15.104:0/3476831778 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98dc1360f0 con 0x7f98dc06de90 2024-03-29T11:22:38.543 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.564+0000 7f98caffd640 10 monclient: _finish_auth 0 2024-03-29T11:22:38.543 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.564+0000 7f98caffd640 10 monclient: _check_auth_tickets 2024-03-29T11:22:38.543 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.564+0000 7f98caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:38.543 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.564+0000 7f98e0ecf640 5 monclient: authenticate success, global_id 4353 2024-03-29T11:22:38.543 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.564+0000 7f98e0ecf640 10 monclient: _renew_subs 2024-03-29T11:22:38.544 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.564+0000 7f98e0ecf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:22:38.544 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.564+0000 7f98e0ecf640 1 -- 172.21.15.104:0/3476831778 --> v1:172.21.15.104:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98dc130050 con 0x7f98dc06de90 2024-03-29T11:22:38.544 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.564+0000 7f98e0ecf640 10 client.4353.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:38.544 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.564+0000 7f98e0ecf640 10 monclient: _renew_subs 2024-03-29T11:22:38.544 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.564+0000 7f98e0ecf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:22:38.544 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.564+0000 7f98e0ecf640 1 -- 172.21.15.104:0/3476831778 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f98dc06e500 con 0x7f98dc06de90 2024-03-29T11:22:38.544 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.564+0000 7f98e0ecf640 1 librados: init done 2024-03-29T11:22:38.546 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.566+0000 7f98c8ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:38.546 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.566+0000 7f98c8ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:38.546 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.566+0000 7f98c8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:22:38.546 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.566+0000 7f98c8ff9640 1 -- 172.21.15.104:0/3476831778 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98c0005080 con 0x7f98dc06de90 2024-03-29T11:22:38.546 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.566+0000 7f98caffd640 1 -- 172.21.15.104:0/3476831778 <== mon.0 v1:172.21.15.104:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98d4004b60 con 0x7f98dc06de90 2024-03-29T11:22:38.546 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.566+0000 7f98caffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:38.547 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.566+0000 7f98caffd640 1 -- 172.21.15.104:0/3476831778 <== mon.0 v1:172.21.15.104:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f98d4004f70 con 0x7f98dc06de90 2024-03-29T11:22:38.547 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.566+0000 7f98caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:38.547 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.566+0000 7f98caffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:38.547 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.566+0000 7f98caffd640 10 monclient: dump: 2024-03-29T11:22:38.547 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:22:38.547 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:22:38.547 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:38.547 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:38.548 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:38.548 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:38.548 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:22:38.548 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:22:38.548 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:22:38.548 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:38.548 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.567+0000 7f98d8d99640 4 set_mon_vals no callback set 2024-03-29T11:22:38.548 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.567+0000 7f98caffd640 1 -- 172.21.15.104:0/3476831778 <== mon.0 v1:172.21.15.104:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f98d4005190 con 0x7f98dc06de90 2024-03-29T11:22:38.548 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.567+0000 7f98caffd640 1 -- 172.21.15.104:0/3476831778 <== mon.0 v1:172.21.15.104:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f98d4084e30 con 0x7f98dc06de90 2024-03-29T11:22:38.549 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.568+0000 7f98da59c640 10 monclient: get_auth_request con 0x7f98bc06dc00 auth_method 0 2024-03-29T11:22:38.549 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.568+0000 7f98caffd640 10 client.4353.objecter ms_dispatch 0x7f98dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:38.549 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.568+0000 7f98caffd640 3 client.4353.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:38.549 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.568+0000 7f98caffd640 3 client.4353.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:38.549 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.568+0000 7f98caffd640 20 client.4353.objecter dump_active .. 0 homeless 2024-03-29T11:22:38.549 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.568+0000 7f98caffd640 10 client.4353.objecter ms_handle_connect 0x7f98bc06dc00 2024-03-29T11:22:38.552 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.575+0000 7f98caffd640 1 -- 172.21.15.104:0/3476831778 <== mon.0 v1:172.21.15.104:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f98d4050870 con 0x7f98dc06de90 2024-03-29T11:22:38.552 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.575+0000 7f98caffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:38.552 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.575+0000 7f98caffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:38.874 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.897+0000 7f98c8ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:38.874 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.897+0000 7f98c8ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:38.874 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.897+0000 7f98c8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:22:38.874 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.897+0000 7f98c8ff9640 1 -- 172.21.15.104:0/3476831778 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98c0005330 con 0x7f98dc06de90 2024-03-29T11:22:38.876 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.900+0000 7f98caffd640 1 -- 172.21.15.104:0/3476831778 <== mon.0 v1:172.21.15.104:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f98d40527a0 con 0x7f98dc06de90 2024-03-29T11:22:38.876 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.900+0000 7f98caffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:38.876 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.900+0000 7f98caffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:38.880 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:22:38.880 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:38.881 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.903+0000 7f98c8ff9640 10 librados: watch_flush enter 2024-03-29T11:22:38.881 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.904+0000 7f98c8ff9640 10 librados: watch_flush exit 2024-03-29T11:22:38.881 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.904+0000 7f98c8ff9640 20 client.4353.objecter shutdown clearing up homeless session... 2024-03-29T11:22:38.881 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.904+0000 7f98c8ff9640 10 client.4353.objecter successfully canceled tick 2024-03-29T11:22:38.881 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.904+0000 7f98c8ff9640 1 -- 172.21.15.104:0/3476831778 >> v1:172.21.15.104:6816/3341126660 conn(0x7f98bc06dc00 legacy=0x7f98bc0700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:38.881 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.904+0000 7f98c8ff9640 10 monclient: shutdown 2024-03-29T11:22:38.881 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.904+0000 7f98c8ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:38.881 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.904+0000 7f98c8ff9640 1 -- 172.21.15.104:0/3476831778 >> v1:172.21.15.104:6789/0 conn(0x7f98dc06de90 legacy=0x7f98dc1e5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:38.881 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.904+0000 7f98c8ff9640 1 -- 172.21.15.104:0/3476831778 shutdown_connections 2024-03-29T11:22:38.881 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.904+0000 7f98c8ff9640 1 -- 172.21.15.104:0/3476831778 >> 172.21.15.104:0/3476831778 conn(0x7f98dc06b660 msgr2=0x7f98dc0a9d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:38.882 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.905+0000 7f98c8ff9640 1 -- 172.21.15.104:0/3476831778 shutdown_connections 2024-03-29T11:22:38.882 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.905+0000 7f98c8ff9640 1 -- 172.21.15.104:0/3476831778 wait complete. 2024-03-29T11:22:38.882 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:38.905+0000 7f98c8ff9640 1 librados: shutdown 2024-03-29T11:22:38.899 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:45.705 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:45.858 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.881+0000 7f23a0b4d640 10 monclient: get_monmap_and_config 2024-03-29T11:22:45.858 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.881+0000 7f23a0b4d640 10 monclient: build_initial_monmap 2024-03-29T11:22:45.858 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.881+0000 7f23a0b4d640 10 monclient: monmap: 2024-03-29T11:22:45.858 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:22:45.858 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:22:45.858 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:22:45.882103+0000 2024-03-29T11:22:45.858 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:22:45.882103+0000 2024-03-29T11:22:45.858 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:45.858 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:45.859 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:22:45.859 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:22:45.859 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:22:45.859 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:45.862 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.885+0000 7f23a0b4d640 1 Processor -- start 2024-03-29T11:22:45.862 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.885+0000 7f23a0b4d640 1 -- start start 2024-03-29T11:22:45.862 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.885+0000 7f23a0b4d640 10 monclient: init 2024-03-29T11:22:45.863 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.886+0000 7f23a0b4d640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:45.863 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.886+0000 7f23a0b4d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:22:45.863 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.886+0000 7f23a0b4d640 10 monclient(hunting): picked mon.noname-a con 0x7f239c0b1f60 addr v1:172.21.15.104:6789/0 2024-03-29T11:22:45.863 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.886+0000 7f23a0b4d640 10 monclient(hunting): picked mon.noname-c con 0x7f239c071eb0 addr v1:172.21.15.119:6789/0 2024-03-29T11:22:45.863 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.886+0000 7f23a0b4d640 10 monclient(hunting): picked mon.noname-b con 0x7f239c143d10 addr v1:172.21.15.104:6790/0 2024-03-29T11:22:45.863 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.886+0000 7f23a0b4d640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f239c06ef80 con 0x7f239c0b1f60 2024-03-29T11:22:45.863 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.886+0000 7f23a0b4d640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f239c06f260 con 0x7f239c071eb0 2024-03-29T11:22:45.863 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.886+0000 7f23a0b4d640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f239c0b40c0 con 0x7f239c143d10 2024-03-29T11:22:45.864 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.886+0000 7f23a0b4d640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:45.864 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.886+0000 7f23a0b4d640 10 monclient(hunting): authenticate will time out at 1147.137554s 2024-03-29T11:22:45.865 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.887+0000 7f239ad9d640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f239c143d10 0x7f239c146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:57784/0 (socket says 172.21.15.104:57784) 2024-03-29T11:22:45.865 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.887+0000 7f239ad9d640 1 -- 172.21.15.104:0/1932498874 learned_addr learned my addr 172.21.15.104:0/1932498874 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:22:45.865 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.887+0000 7f239959a640 1 -- 172.21.15.104:0/1932498874 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 315606441 0 0) 0x7f239c0b40c0 con 0x7f239c143d10 2024-03-29T11:22:45.865 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.887+0000 7f239959a640 10 monclient(hunting): _negotiate 2024-03-29T11:22:45.865 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.887+0000 7f239959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:45.865 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.887+0000 7f239959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:45.865 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.887+0000 7f239959a640 10 monclient(hunting): my global_id is 4361 2024-03-29T11:22:45.865 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.887+0000 7f239959a640 1 -- 172.21.15.104:0/1932498874 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2384003520 con 0x7f239c143d10 2024-03-29T11:22:45.865 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.888+0000 7f239959a640 1 -- 172.21.15.104:0/1932498874 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2591994342 0 0) 0x7f2384003520 con 0x7f239c143d10 2024-03-29T11:22:45.866 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.888+0000 7f239959a640 1 -- 172.21.15.104:0/1932498874 >> v1:172.21.15.119:6789/0 conn(0x7f239c071eb0 legacy=0x7f239c146470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:45.866 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.888+0000 7f239959a640 1 -- 172.21.15.104:0/1932498874 >> v1:172.21.15.104:6789/0 conn(0x7f239c0b1f60 legacy=0x7f239c143600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:45.868 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.890+0000 7f239959a640 10 monclient: _finish_hunting 0 2024-03-29T11:22:45.868 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.890+0000 7f239959a640 1 monclient: found mon.noname-b 2024-03-29T11:22:45.868 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.890+0000 7f239959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:45.868 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.890+0000 7f239959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.104:6790/0 2024-03-29T11:22:45.868 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.890+0000 7f239959a640 1 -- 172.21.15.104:0/1932498874 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f239c155d70 con 0x7f239c143d10 2024-03-29T11:22:45.868 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.890+0000 7f239959a640 10 monclient: _finish_auth 0 2024-03-29T11:22:45.868 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.890+0000 7f239959a640 10 monclient: _check_auth_tickets 2024-03-29T11:22:45.869 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.890+0000 7f239959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:45.869 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.890+0000 7f239959a640 1 -- 172.21.15.104:0/1932498874 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f238c001690 con 0x7f239c143d10 2024-03-29T11:22:45.869 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.890+0000 7f239959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:45.869 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.890+0000 7f239959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:45.869 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.890+0000 7f239959a640 10 monclient: dump: 2024-03-29T11:22:45.869 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:22:45.869 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:22:45.869 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:45.869 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:45.869 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:45.869 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:45.869 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:22:45.869 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:22:45.870 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:22:45.870 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:45.870 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.890+0000 7f23a0b4d640 5 monclient: authenticate success, global_id 4361 2024-03-29T11:22:45.870 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.890+0000 7f23a0b4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:45.871 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.891+0000 7f239959a640 1 -- 172.21.15.104:0/1932498874 <== mon.2 v1:172.21.15.104:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f238c001860 con 0x7f239c143d10 2024-03-29T11:22:45.871 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.891+0000 7f239959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:45.871 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.891+0000 7f239959a640 1 -- 172.21.15.104:0/1932498874 <== mon.2 v1:172.21.15.104:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f238c001cb0 con 0x7f239c143d10 2024-03-29T11:22:45.871 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.891+0000 7f239959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:45.871 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.891+0000 7f239959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:45.871 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.891+0000 7f239959a640 10 monclient: dump: 2024-03-29T11:22:45.871 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:22:45.871 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:22:45.871 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:45.871 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:45.871 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:45.871 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:45.872 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:22:45.872 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:22:45.872 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:22:45.872 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:45.872 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.892+0000 7f23a0b4d640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:45.872 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.892+0000 7f23a0b4d640 4 set_mon_vals no callback set 2024-03-29T11:22:45.872 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.892+0000 7f23a0b4d640 10 monclient: shutdown 2024-03-29T11:22:45.872 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.892+0000 7f23a0b4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:45.872 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.892+0000 7f23a0b4d640 1 -- 172.21.15.104:0/1932498874 >> v1:172.21.15.104:6790/0 conn(0x7f239c143d10 legacy=0x7f239c146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:45.872 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.893+0000 7f23a0b4d640 1 -- 172.21.15.104:0/1932498874 shutdown_connections 2024-03-29T11:22:45.872 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.893+0000 7f23a0b4d640 1 -- 172.21.15.104:0/1932498874 >> 172.21.15.104:0/1932498874 conn(0x7f239c066a00 msgr2=0x7f239c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:45.872 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.893+0000 7f23a0b4d640 1 -- 172.21.15.104:0/1932498874 shutdown_connections 2024-03-29T11:22:45.872 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.893+0000 7f23a0b4d640 1 -- 172.21.15.104:0/1932498874 wait complete. 2024-03-29T11:22:45.873 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.894+0000 7f23a0b4d640 10 monclient: build_initial_monmap 2024-03-29T11:22:45.873 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.894+0000 7f23a0b4d640 10 monclient: monmap: 2024-03-29T11:22:45.873 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:22:45.873 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:45.873 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:22:45.873 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:22:45.873 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:45.873 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:45.873 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:22:45.873 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:22:45.873 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:22:45.873 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:45.873 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.894+0000 7f23a0b4d640 1 librados: starting msgr at 2024-03-29T11:22:45.874 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.894+0000 7f23a0b4d640 1 librados: starting objecter 2024-03-29T11:22:45.874 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.894+0000 7f23a0b4d640 1 Processor -- start 2024-03-29T11:22:45.874 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.894+0000 7f23a0b4d640 1 -- start start 2024-03-29T11:22:45.874 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.894+0000 7f23a0b4d640 1 librados: setting wanted keys 2024-03-29T11:22:45.874 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.894+0000 7f23a0b4d640 1 librados: calling monclient init 2024-03-29T11:22:45.874 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.894+0000 7f23a0b4d640 10 monclient: init 2024-03-29T11:22:45.874 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.895+0000 7f23a0b4d640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:45.874 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.895+0000 7f23a0b4d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:22:45.874 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.895+0000 7f23a0b4d640 10 monclient(hunting): picked mon.noname-c con 0x7f239c0b1f60 addr v1:172.21.15.104:6790/0 2024-03-29T11:22:45.875 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.898+0000 7f239a59c640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f239c0b1f60 0x7f239c131c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:58174/0 (socket says 172.21.15.104:58174) 2024-03-29T11:22:45.875 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.898+0000 7f23a0b4d640 10 monclient(hunting): picked mon.noname-b con 0x7f239c143d10 addr v1:172.21.15.119:6789/0 2024-03-29T11:22:45.875 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.898+0000 7f23a0b4d640 10 monclient(hunting): picked mon.noname-a con 0x7f239c1337d0 addr v1:172.21.15.104:6789/0 2024-03-29T11:22:45.876 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.898+0000 7f2399d9b640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f239c143d10 0x7f239c141190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:47775/0 (socket says 172.21.15.104:47775) 2024-03-29T11:22:45.876 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.898+0000 7f2399d9b640 1 -- 172.21.15.104:0/883902409 learned_addr learned my addr 172.21.15.104:0/883902409 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:22:45.876 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.898+0000 7f23a0b4d640 1 -- 172.21.15.104:0/883902409 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f239c153be0 con 0x7f239c1337d0 2024-03-29T11:22:45.876 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.898+0000 7f23a0b4d640 1 -- 172.21.15.104:0/883902409 --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f239c154940 con 0x7f239c143d10 2024-03-29T11:22:45.876 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.898+0000 7f23a0b4d640 1 -- 172.21.15.104:0/883902409 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f239c154c20 con 0x7f239c0b1f60 2024-03-29T11:22:45.876 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.898+0000 7f23a0b4d640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:45.876 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.898+0000 7f23a0b4d640 10 monclient(hunting): authenticate will time out at 1147.149645s 2024-03-29T11:22:45.876 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.898+0000 7f238affd640 10 client.?.objecter ms_handle_connect 0x7f239c143d10 2024-03-29T11:22:45.876 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.898+0000 7f238affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:45.876 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.898+0000 7f238affd640 10 client.?.objecter ms_handle_connect 0x7f239c0b1f60 2024-03-29T11:22:45.876 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.898+0000 7f238affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:45.876 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.898+0000 7f238affd640 1 -- 172.21.15.104:0/883902409 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2408705637 0 0) 0x7f239c154940 con 0x7f239c143d10 2024-03-29T11:22:45.876 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.899+0000 7f238affd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:45.876 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.899+0000 7f238affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:45.877 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.899+0000 7f238affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:45.877 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.899+0000 7f238affd640 10 monclient(hunting): my global_id is 4327 2024-03-29T11:22:45.877 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.899+0000 7f238affd640 1 -- 172.21.15.104:0/883902409 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f237c004eb0 con 0x7f239c143d10 2024-03-29T11:22:45.878 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.899+0000 7f238affd640 1 -- 172.21.15.104:0/883902409 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3540085904 0 0) 0x7f239c154c20 con 0x7f239c0b1f60 2024-03-29T11:22:45.878 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.899+0000 7f238affd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:45.878 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.899+0000 7f238affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:45.878 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.899+0000 7f238affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:45.878 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.899+0000 7f238affd640 10 monclient(hunting): my global_id is 4364 2024-03-29T11:22:45.878 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.899+0000 7f238affd640 1 -- 172.21.15.104:0/883902409 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f237c0051d0 con 0x7f239c0b1f60 2024-03-29T11:22:45.878 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.899+0000 7f238affd640 1 -- 172.21.15.104:0/883902409 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3612801662 0 0) 0x7f237c004eb0 con 0x7f239c143d10 2024-03-29T11:22:45.878 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.899+0000 7f238affd640 1 -- 172.21.15.104:0/883902409 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f237c005640 con 0x7f239c143d10 2024-03-29T11:22:45.878 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.899+0000 7f238affd640 1 -- 172.21.15.104:0/883902409 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f2394001bb0 con 0x7f239c143d10 2024-03-29T11:22:45.878 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.899+0000 7f238affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:45.878 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.899+0000 7f238affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:45.878 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.899+0000 7f238affd640 10 monclient(hunting): dump: 2024-03-29T11:22:45.879 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:22:45.879 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:22:45.879 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:45.879 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:45.879 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:45.879 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:45.879 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:22:45.879 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:22:45.879 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:22:45.879 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:45.879 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.899+0000 7f238affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:45.879 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.899+0000 7f238affd640 1 -- 172.21.15.104:0/883902409 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1064929201 0 0) 0x7f237c0051d0 con 0x7f239c0b1f60 2024-03-29T11:22:45.879 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.899+0000 7f238affd640 1 -- 172.21.15.104:0/883902409 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f237c005d30 con 0x7f239c0b1f60 2024-03-29T11:22:45.879 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.899+0000 7f238affd640 1 -- 172.21.15.104:0/883902409 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f2390003f40 con 0x7f239c0b1f60 2024-03-29T11:22:45.879 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.899+0000 7f238affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:45.880 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.899+0000 7f238affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:45.880 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.899+0000 7f238affd640 10 monclient(hunting): dump: 2024-03-29T11:22:45.880 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:22:45.880 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:22:45.880 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:45.880 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:45.880 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:45.880 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:45.880 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:22:45.880 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:22:45.880 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:22:45.880 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:45.880 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.899+0000 7f238affd640 1 -- 172.21.15.104:0/883902409 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2524306959 0 0) 0x7f237c005640 con 0x7f239c143d10 2024-03-29T11:22:45.880 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.899+0000 7f238affd640 1 -- 172.21.15.104:0/883902409 >> v1:172.21.15.104:6790/0 conn(0x7f239c0b1f60 legacy=0x7f239c131c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:45.880 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.899+0000 7f238affd640 1 -- 172.21.15.104:0/883902409 >> v1:172.21.15.104:6789/0 conn(0x7f239c1337d0 legacy=0x7f239c1418a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:22:45.884 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.901+0000 7f238affd640 10 monclient: _finish_hunting 0 2024-03-29T11:22:45.884 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.901+0000 7f238affd640 1 monclient: found mon.b 2024-03-29T11:22:45.884 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.901+0000 7f238affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:45.884 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.901+0000 7f238affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:22:45.884 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.901+0000 7f238affd640 1 -- 172.21.15.104:0/883902409 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f239c154f00 con 0x7f239c143d10 2024-03-29T11:22:45.884 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.901+0000 7f238affd640 10 monclient: _finish_auth 0 2024-03-29T11:22:45.884 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.901+0000 7f238affd640 10 monclient: _check_auth_tickets 2024-03-29T11:22:45.884 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.901+0000 7f238affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:45.884 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.902+0000 7f23a0b4d640 5 monclient: authenticate success, global_id 4327 2024-03-29T11:22:45.885 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.902+0000 7f23a0b4d640 10 monclient: _renew_subs 2024-03-29T11:22:45.885 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.902+0000 7f23a0b4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:22:45.885 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.902+0000 7f23a0b4d640 1 -- 172.21.15.104:0/883902409 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f239c142620 con 0x7f239c143d10 2024-03-29T11:22:45.885 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.902+0000 7f238affd640 1 -- 172.21.15.104:0/883902409 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23940020c0 con 0x7f239c143d10 2024-03-29T11:22:45.885 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.902+0000 7f238affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:45.885 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.902+0000 7f238affd640 1 -- 172.21.15.104:0/883902409 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f2394001bb0 con 0x7f239c143d10 2024-03-29T11:22:45.885 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.902+0000 7f238affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:45.885 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.902+0000 7f238affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:45.885 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.902+0000 7f238affd640 10 monclient: dump: 2024-03-29T11:22:45.885 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:22:45.885 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:22:45.885 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:45.885 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:45.885 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:45.885 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:45.885 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:22:45.886 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:22:45.886 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:22:45.886 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:45.886 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.902+0000 7f238bfff640 4 set_mon_vals no callback set 2024-03-29T11:22:45.886 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.903+0000 7f23a0b4d640 10 client.4327.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:45.886 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.903+0000 7f23a0b4d640 10 monclient: _renew_subs 2024-03-29T11:22:45.886 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.903+0000 7f23a0b4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:22:45.886 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.903+0000 7f23a0b4d640 1 -- 172.21.15.104:0/883902409 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f239c132380 con 0x7f239c143d10 2024-03-29T11:22:45.886 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.903+0000 7f23a0b4d640 1 librados: init done 2024-03-29T11:22:45.886 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.904+0000 7f23a0b4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:45.886 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.904+0000 7f23a0b4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:45.886 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.904+0000 7f23a0b4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:22:45.886 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.904+0000 7f23a0b4d640 1 -- 172.21.15.104:0/883902409 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f239c155500 con 0x7f239c143d10 2024-03-29T11:22:45.886 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.905+0000 7f238affd640 1 -- 172.21.15.104:0/883902409 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f239401ab90 con 0x7f239c143d10 2024-03-29T11:22:45.886 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.906+0000 7f238affd640 1 -- 172.21.15.104:0/883902409 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f2394054930 con 0x7f239c143d10 2024-03-29T11:22:45.886 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.906+0000 7f238affd640 10 client.4327.objecter ms_dispatch 0x7f239c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:45.887 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.906+0000 7f238affd640 3 client.4327.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:45.887 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.906+0000 7f238affd640 3 client.4327.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:45.887 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.906+0000 7f238affd640 20 client.4327.objecter dump_active .. 0 homeless 2024-03-29T11:22:45.887 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.906+0000 7f239a59c640 10 monclient: get_auth_request con 0x7f237c06e820 auth_method 0 2024-03-29T11:22:45.887 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.906+0000 7f238affd640 10 client.4327.objecter ms_handle_connect 0x7f237c06e820 2024-03-29T11:22:45.890 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.913+0000 7f238affd640 1 -- 172.21.15.104:0/883902409 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2394050bd0 con 0x7f239c143d10 2024-03-29T11:22:45.890 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.913+0000 7f238affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:45.890 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:45.913+0000 7f238affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:46.211 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:46.234+0000 7f23a0b4d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:46.211 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:46.234+0000 7f23a0b4d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:46.211 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:46.234+0000 7f23a0b4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:22:46.211 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:46.234+0000 7f23a0b4d640 1 -- 172.21.15.104:0/883902409 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f239c075f10 con 0x7f239c143d10 2024-03-29T11:22:46.212 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:46.234+0000 7f238affd640 1 -- 172.21.15.104:0/883902409 <== mon.1 v1:172.21.15.119:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f2394050bd0 con 0x7f239c143d10 2024-03-29T11:22:46.212 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:46.234+0000 7f238affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:46.212 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:46.234+0000 7f238affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:46.212 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:22:46.212 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:46.216 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:46.239+0000 7f23a0b4d640 10 librados: watch_flush enter 2024-03-29T11:22:46.216 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:46.239+0000 7f23a0b4d640 10 librados: watch_flush exit 2024-03-29T11:22:46.216 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:46.239+0000 7f23a0b4d640 20 client.4327.objecter shutdown clearing up homeless session... 2024-03-29T11:22:46.216 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:46.239+0000 7f23a0b4d640 10 client.4327.objecter successfully canceled tick 2024-03-29T11:22:46.217 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:46.239+0000 7f23a0b4d640 1 -- 172.21.15.104:0/883902409 >> v1:172.21.15.104:6816/3341126660 conn(0x7f237c06e820 legacy=0x7f237c070ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:46.217 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:46.239+0000 7f23a0b4d640 10 monclient: shutdown 2024-03-29T11:22:46.217 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:46.239+0000 7f23a0b4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:46.217 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:46.239+0000 7f23a0b4d640 1 -- 172.21.15.104:0/883902409 >> v1:172.21.15.119:6789/0 conn(0x7f239c143d10 legacy=0x7f239c141190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:46.217 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:46.240+0000 7f23a0b4d640 1 -- 172.21.15.104:0/883902409 shutdown_connections 2024-03-29T11:22:46.217 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:46.240+0000 7f23a0b4d640 1 -- 172.21.15.104:0/883902409 >> 172.21.15.104:0/883902409 conn(0x7f239c066a00 msgr2=0x7f239c0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:46.217 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:46.240+0000 7f23a0b4d640 1 -- 172.21.15.104:0/883902409 shutdown_connections 2024-03-29T11:22:46.217 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:46.240+0000 7f23a0b4d640 1 -- 172.21.15.104:0/883902409 wait complete. 2024-03-29T11:22:46.218 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:46.241+0000 7f23a0b4d640 1 librados: shutdown 2024-03-29T11:22:46.235 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:53.042 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:53.188 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.211+0000 7fc455b89640 10 monclient: get_monmap_and_config 2024-03-29T11:22:53.188 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.211+0000 7fc455b89640 10 monclient: build_initial_monmap 2024-03-29T11:22:53.188 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.211+0000 7fc455b89640 10 monclient: monmap: 2024-03-29T11:22:53.188 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:22:53.189 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:22:53.189 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:22:53.212153+0000 2024-03-29T11:22:53.189 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:22:53.212153+0000 2024-03-29T11:22:53.189 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:53.189 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:53.189 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:22:53.189 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:22:53.189 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:22:53.189 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:53.190 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.212+0000 7fc455b89640 1 Processor -- start 2024-03-29T11:22:53.190 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.212+0000 7fc455b89640 1 -- start start 2024-03-29T11:22:53.190 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.212+0000 7fc455b89640 10 monclient: init 2024-03-29T11:22:53.191 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.213+0000 7fc455b89640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:53.191 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.213+0000 7fc455b89640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:22:53.192 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.213+0000 7fc455b89640 10 monclient(hunting): picked mon.noname-a con 0x7fc4500b0860 addr v1:172.21.15.104:6789/0 2024-03-29T11:22:53.192 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.214+0000 7fc455b89640 10 monclient(hunting): picked mon.noname-b con 0x7fc45013ef30 addr v1:172.21.15.104:6790/0 2024-03-29T11:22:53.192 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.214+0000 7fc455b89640 10 monclient(hunting): picked mon.noname-c con 0x7fc4500af8e0 addr v1:172.21.15.119:6789/0 2024-03-29T11:22:53.192 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.214+0000 7fc455b89640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4500b0c40 con 0x7fc4500b0860 2024-03-29T11:22:53.192 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.214+0000 7fc455b89640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4500aed10 con 0x7fc4500af8e0 2024-03-29T11:22:53.192 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.214+0000 7fc44f7fe640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fc4500b0860 0x7fc45013e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34526/0 (socket says 172.21.15.104:34526) 2024-03-29T11:22:53.192 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.214+0000 7fc44effd640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7fc45013ef30 0x7fc4500af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:33914/0 (socket says 172.21.15.104:33914) 2024-03-29T11:22:53.192 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.214+0000 7fc44effd640 1 -- 172.21.15.104:0/2819238295 learned_addr learned my addr 172.21.15.104:0/2819238295 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:22:53.192 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.214+0000 7fc455b89640 1 -- 172.21.15.104:0/2819238295 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4500afd50 con 0x7fc45013ef30 2024-03-29T11:22:53.192 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.214+0000 7fc455b89640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:53.192 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.214+0000 7fc455b89640 10 monclient(hunting): authenticate will time out at 1154.465718s 2024-03-29T11:22:53.194 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.215+0000 7fc44e7fc640 1 -- 172.21.15.104:0/2819238295 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 674939315 0 0) 0x7fc4500aed10 con 0x7fc4500af8e0 2024-03-29T11:22:53.194 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.215+0000 7fc44e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:53.194 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.215+0000 7fc44e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:53.194 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.215+0000 7fc44e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:53.194 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.215+0000 7fc44e7fc640 10 monclient(hunting): my global_id is 4330 2024-03-29T11:22:53.194 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.215+0000 7fc44e7fc640 1 -- 172.21.15.104:0/2819238295 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc438003520 con 0x7fc4500af8e0 2024-03-29T11:22:53.194 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.215+0000 7fc44e7fc640 1 -- 172.21.15.104:0/2819238295 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1981626932 0 0) 0x7fc4500afd50 con 0x7fc45013ef30 2024-03-29T11:22:53.194 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.215+0000 7fc44e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:53.194 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.215+0000 7fc44e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:53.194 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.215+0000 7fc44e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:53.194 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.215+0000 7fc44e7fc640 10 monclient(hunting): my global_id is 4367 2024-03-29T11:22:53.194 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.215+0000 7fc44e7fc640 1 -- 172.21.15.104:0/2819238295 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4500aed10 con 0x7fc45013ef30 2024-03-29T11:22:53.195 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.215+0000 7fc44e7fc640 1 -- 172.21.15.104:0/2819238295 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1210793126 0 0) 0x7fc438003520 con 0x7fc4500af8e0 2024-03-29T11:22:53.195 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.215+0000 7fc44e7fc640 1 -- 172.21.15.104:0/2819238295 >> v1:172.21.15.104:6790/0 conn(0x7fc45013ef30 legacy=0x7fc4500af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:53.195 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.215+0000 7fc44e7fc640 1 -- 172.21.15.104:0/2819238295 >> v1:172.21.15.104:6789/0 conn(0x7fc4500b0860 legacy=0x7fc45013e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:53.195 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.215+0000 7fc44e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:22:53.195 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.215+0000 7fc44e7fc640 1 monclient: found mon.noname-c 2024-03-29T11:22:53.195 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.215+0000 7fc44e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:53.195 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.215+0000 7fc44e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.119:6789/0 2024-03-29T11:22:53.195 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.215+0000 7fc44e7fc640 1 -- 172.21.15.104:0/2819238295 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4500ab1b0 con 0x7fc4500af8e0 2024-03-29T11:22:53.195 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.215+0000 7fc44e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:22:53.195 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.215+0000 7fc44e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:22:53.195 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.215+0000 7fc44e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:53.195 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.215+0000 7fc44e7fc640 1 -- 172.21.15.104:0/2819238295 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fc448001a10 con 0x7fc4500af8e0 2024-03-29T11:22:53.195 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.215+0000 7fc44e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:53.195 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.215+0000 7fc44e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:53.195 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.216+0000 7fc44e7fc640 10 monclient: dump: 2024-03-29T11:22:53.195 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:22:53.196 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:22:53.196 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:53.196 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:53.196 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:53.196 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:53.196 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:22:53.196 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:22:53.196 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:22:53.196 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:53.196 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.216+0000 7fc455b89640 5 monclient: authenticate success, global_id 4330 2024-03-29T11:22:53.196 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.216+0000 7fc455b89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:53.196 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.216+0000 7fc44e7fc640 1 -- 172.21.15.104:0/2819238295 <== mon.1 v1:172.21.15.119:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc448001c30 con 0x7fc4500af8e0 2024-03-29T11:22:53.196 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.216+0000 7fc44e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:53.196 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.216+0000 7fc44e7fc640 1 -- 172.21.15.104:0/2819238295 <== mon.1 v1:172.21.15.119:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fc448001fc0 con 0x7fc4500af8e0 2024-03-29T11:22:53.196 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.216+0000 7fc455b89640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:53.196 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.216+0000 7fc455b89640 4 set_mon_vals no callback set 2024-03-29T11:22:53.197 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.216+0000 7fc44e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:53.197 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.216+0000 7fc44e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:53.197 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.216+0000 7fc44e7fc640 10 monclient: dump: 2024-03-29T11:22:53.197 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:22:53.197 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:22:53.197 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:53.197 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:53.197 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:53.197 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:53.197 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:22:53.197 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:22:53.197 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:22:53.197 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:53.205 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.217+0000 7fc455b89640 10 monclient: shutdown 2024-03-29T11:22:53.205 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.217+0000 7fc455b89640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:53.205 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.217+0000 7fc455b89640 1 -- 172.21.15.104:0/2819238295 >> v1:172.21.15.119:6789/0 conn(0x7fc4500af8e0 legacy=0x7fc450149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:53.205 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.217+0000 7fc455b89640 1 -- 172.21.15.104:0/2819238295 shutdown_connections 2024-03-29T11:22:53.205 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.217+0000 7fc455b89640 1 -- 172.21.15.104:0/2819238295 >> 172.21.15.104:0/2819238295 conn(0x7fc45006b660 msgr2=0x7fc45006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:53.205 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.218+0000 7fc455b89640 1 -- 172.21.15.104:0/2819238295 shutdown_connections 2024-03-29T11:22:53.205 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.218+0000 7fc455b89640 1 -- 172.21.15.104:0/2819238295 wait complete. 2024-03-29T11:22:53.205 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.218+0000 7fc455b89640 10 monclient: build_initial_monmap 2024-03-29T11:22:53.205 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.218+0000 7fc455b89640 10 monclient: monmap: 2024-03-29T11:22:53.205 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:22:53.205 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:53.206 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:22:53.206 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:22:53.206 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:53.206 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:53.206 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:22:53.206 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:22:53.206 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:22:53.206 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:53.206 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.219+0000 7fc455b89640 1 librados: starting msgr at 2024-03-29T11:22:53.206 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.219+0000 7fc455b89640 1 librados: starting objecter 2024-03-29T11:22:53.206 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.219+0000 7fc455b89640 1 Processor -- start 2024-03-29T11:22:53.206 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.219+0000 7fc455b89640 1 -- start start 2024-03-29T11:22:53.206 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.219+0000 7fc455b89640 1 librados: setting wanted keys 2024-03-29T11:22:53.206 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.219+0000 7fc455b89640 1 librados: calling monclient init 2024-03-29T11:22:53.206 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.219+0000 7fc455b89640 10 monclient: init 2024-03-29T11:22:53.206 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.220+0000 7fc455b89640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:53.207 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.220+0000 7fc455b89640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:22:53.207 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.220+0000 7fc455b89640 10 monclient(hunting): picked mon.noname-c con 0x7fc4500af8e0 addr v1:172.21.15.104:6790/0 2024-03-29T11:22:53.207 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.220+0000 7fc455b89640 10 monclient(hunting): picked mon.noname-b con 0x7fc4500b0860 addr v1:172.21.15.119:6789/0 2024-03-29T11:22:53.207 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.220+0000 7fc455b89640 10 monclient(hunting): picked mon.noname-a con 0x7fc45013ef30 addr v1:172.21.15.104:6789/0 2024-03-29T11:22:53.207 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.220+0000 7fc455b89640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4501cc3e0 con 0x7fc45013ef30 2024-03-29T11:22:53.207 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.220+0000 7fc455b89640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4501cd4a0 con 0x7fc4500b0860 2024-03-29T11:22:53.207 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.220+0000 7fc455b89640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4501cd780 con 0x7fc4500af8e0 2024-03-29T11:22:53.207 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.220+0000 7fc455b89640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:53.207 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.220+0000 7fc455b89640 10 monclient(hunting): authenticate will time out at 1154.471475s 2024-03-29T11:22:53.207 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.220+0000 7fc44ffff640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fc45013ef30 0x7fc4501e7380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:34724/0 (socket says 172.21.15.104:34724) 2024-03-29T11:22:53.207 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.220+0000 7fc44ffff640 1 -- 172.21.15.104:0/1797746062 learned_addr learned my addr 172.21.15.104:0/1797746062 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:22:53.207 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.221+0000 7fc437fff640 10 client.?.objecter ms_handle_connect 0x7fc4500af8e0 2024-03-29T11:22:53.207 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.221+0000 7fc437fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:53.207 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.221+0000 7fc437fff640 10 client.?.objecter ms_handle_connect 0x7fc45013ef30 2024-03-29T11:22:53.207 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.221+0000 7fc437fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:53.208 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.221+0000 7fc437fff640 10 client.?.objecter ms_handle_connect 0x7fc4500b0860 2024-03-29T11:22:53.208 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.221+0000 7fc437fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:53.208 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.221+0000 7fc437fff640 1 -- 172.21.15.104:0/1797746062 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1051956391 0 0) 0x7fc4501cc3e0 con 0x7fc45013ef30 2024-03-29T11:22:53.208 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.221+0000 7fc437fff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:53.208 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.221+0000 7fc437fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:53.208 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.221+0000 7fc437fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:53.208 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.221+0000 7fc437fff640 10 monclient(hunting): my global_id is 4359 2024-03-29T11:22:53.208 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.221+0000 7fc437fff640 1 -- 172.21.15.104:0/1797746062 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc43c004eb0 con 0x7fc45013ef30 2024-03-29T11:22:53.208 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.221+0000 7fc437fff640 1 -- 172.21.15.104:0/1797746062 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1463208387 0 0) 0x7fc4501cd4a0 con 0x7fc4500b0860 2024-03-29T11:22:53.208 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.221+0000 7fc437fff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:53.208 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.221+0000 7fc437fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:53.208 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.221+0000 7fc437fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:53.208 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.221+0000 7fc437fff640 10 monclient(hunting): my global_id is 4333 2024-03-29T11:22:53.208 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.221+0000 7fc437fff640 1 -- 172.21.15.104:0/1797746062 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4501cc3e0 con 0x7fc4500b0860 2024-03-29T11:22:53.208 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.222+0000 7fc437fff640 1 -- 172.21.15.104:0/1797746062 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4125157363 0 0) 0x7fc4501cd780 con 0x7fc4500af8e0 2024-03-29T11:22:53.208 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.222+0000 7fc437fff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:53.209 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.222+0000 7fc437fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:53.209 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.222+0000 7fc437fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:53.209 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.222+0000 7fc437fff640 10 monclient(hunting): my global_id is 4370 2024-03-29T11:22:53.209 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.222+0000 7fc437fff640 1 -- 172.21.15.104:0/1797746062 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4501cd4a0 con 0x7fc4500af8e0 2024-03-29T11:22:53.209 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.222+0000 7fc437fff640 1 -- 172.21.15.104:0/1797746062 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2737870590 0 0) 0x7fc43c004eb0 con 0x7fc45013ef30 2024-03-29T11:22:53.209 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.222+0000 7fc437fff640 1 -- 172.21.15.104:0/1797746062 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4501cd780 con 0x7fc45013ef30 2024-03-29T11:22:53.209 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.222+0000 7fc437fff640 1 -- 172.21.15.104:0/1797746062 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2114001397 0 0) 0x7fc4501cc3e0 con 0x7fc4500b0860 2024-03-29T11:22:53.209 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.222+0000 7fc437fff640 1 -- 172.21.15.104:0/1797746062 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc43c004eb0 con 0x7fc4500b0860 2024-03-29T11:22:53.209 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.222+0000 7fc437fff640 1 -- 172.21.15.104:0/1797746062 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fc448001a10 con 0x7fc45013ef30 2024-03-29T11:22:53.209 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.222+0000 7fc437fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:53.209 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.222+0000 7fc437fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:53.209 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.222+0000 7fc437fff640 10 monclient(hunting): dump: 2024-03-29T11:22:53.209 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:22:53.209 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:22:53.209 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:53.209 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:53.209 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:53.210 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:53.210 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:22:53.210 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:22:53.210 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:22:53.210 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:53.210 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.222+0000 7fc437fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:53.210 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.222+0000 7fc437fff640 1 -- 172.21.15.104:0/1797746062 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fc444001730 con 0x7fc4500b0860 2024-03-29T11:22:53.210 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.222+0000 7fc437fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:53.210 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.222+0000 7fc437fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:53.210 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.222+0000 7fc437fff640 10 monclient(hunting): dump: 2024-03-29T11:22:53.210 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:22:53.210 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:22:53.210 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:53.210 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:53.211 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:53.211 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:53.211 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:22:53.211 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:22:53.211 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:22:53.211 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:53.211 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.222+0000 7fc437fff640 1 -- 172.21.15.104:0/1797746062 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3826382496 0 0) 0x7fc43c004eb0 con 0x7fc4500b0860 2024-03-29T11:22:53.211 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.222+0000 7fc437fff640 1 -- 172.21.15.104:0/1797746062 >> v1:172.21.15.104:6790/0 conn(0x7fc4500af8e0 legacy=0x7fc4501c5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:53.211 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.222+0000 7fc437fff640 1 -- 172.21.15.104:0/1797746062 >> v1:172.21.15.104:6789/0 conn(0x7fc45013ef30 legacy=0x7fc4501e7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:53.211 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.222+0000 7fc437fff640 10 monclient: _finish_hunting 0 2024-03-29T11:22:53.211 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.222+0000 7fc437fff640 1 monclient: found mon.b 2024-03-29T11:22:53.211 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.222+0000 7fc437fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:53.211 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.223+0000 7fc437fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:22:53.211 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.223+0000 7fc437fff640 1 -- 172.21.15.104:0/1797746062 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4501cc870 con 0x7fc4500b0860 2024-03-29T11:22:53.212 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.223+0000 7fc437fff640 10 monclient: _finish_auth 0 2024-03-29T11:22:53.212 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.223+0000 7fc437fff640 10 monclient: _check_auth_tickets 2024-03-29T11:22:53.212 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.223+0000 7fc437fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:53.212 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.223+0000 7fc455b89640 5 monclient: authenticate success, global_id 4333 2024-03-29T11:22:53.212 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.223+0000 7fc455b89640 10 monclient: _renew_subs 2024-03-29T11:22:53.212 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.223+0000 7fc455b89640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:22:53.212 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.223+0000 7fc455b89640 1 -- 172.21.15.104:0/1797746062 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4501c5b50 con 0x7fc4500b0860 2024-03-29T11:22:53.212 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.223+0000 7fc455b89640 10 client.4333.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:53.212 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.223+0000 7fc455b89640 10 monclient: _renew_subs 2024-03-29T11:22:53.212 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.223+0000 7fc455b89640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:22:53.212 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.223+0000 7fc455b89640 1 -- 172.21.15.104:0/1797746062 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4501cbd50 con 0x7fc4500b0860 2024-03-29T11:22:53.212 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.223+0000 7fc455b89640 1 librados: init done 2024-03-29T11:22:53.212 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.223+0000 7fc437fff640 1 -- 172.21.15.104:0/1797746062 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc444001ca0 con 0x7fc4500b0860 2024-03-29T11:22:53.212 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.223+0000 7fc437fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:53.212 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.223+0000 7fc437fff640 1 -- 172.21.15.104:0/1797746062 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fc444004980 con 0x7fc4500b0860 2024-03-29T11:22:53.213 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.223+0000 7fc44dffb640 4 set_mon_vals no callback set 2024-03-29T11:22:53.213 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.223+0000 7fc437fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:53.213 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.224+0000 7fc437fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:53.213 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.224+0000 7fc437fff640 10 monclient: dump: 2024-03-29T11:22:53.213 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:22:53.213 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:22:53.213 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:53.213 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:22:53.213 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:53.213 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:22:53.213 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:22:53.213 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:22:53.213 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:22:53.213 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:22:53.213 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.224+0000 7fc455b89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:53.213 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.225+0000 7fc455b89640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:53.214 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.225+0000 7fc455b89640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:22:53.214 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.225+0000 7fc455b89640 1 -- 172.21.15.104:0/1797746062 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc430005080 con 0x7fc4500b0860 2024-03-29T11:22:53.214 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.225+0000 7fc437fff640 1 -- 172.21.15.104:0/1797746062 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7fc444004b60 con 0x7fc4500b0860 2024-03-29T11:22:53.214 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.226+0000 7fc44f7fe640 10 monclient: get_auth_request con 0x7fc43c06dfe0 auth_method 0 2024-03-29T11:22:53.214 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.226+0000 7fc437fff640 1 -- 172.21.15.104:0/1797746062 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7fc4440848d0 con 0x7fc4500b0860 2024-03-29T11:22:53.214 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.226+0000 7fc437fff640 10 client.4333.objecter ms_dispatch 0x7fc450000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:53.214 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.226+0000 7fc437fff640 3 client.4333.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:53.214 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.226+0000 7fc437fff640 3 client.4333.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:53.214 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.226+0000 7fc437fff640 20 client.4333.objecter dump_active .. 0 homeless 2024-03-29T11:22:53.214 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.226+0000 7fc437fff640 10 client.4333.objecter ms_handle_connect 0x7fc43c06dfe0 2024-03-29T11:22:53.214 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.234+0000 7fc437fff640 1 -- 172.21.15.104:0/1797746062 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc4440501b0 con 0x7fc4500b0860 2024-03-29T11:22:53.215 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.234+0000 7fc437fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:53.215 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.234+0000 7fc437fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:53.558 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.580+0000 7fc455b89640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:53.558 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.581+0000 7fc455b89640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:53.558 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.581+0000 7fc455b89640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:22:53.558 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.581+0000 7fc455b89640 1 -- 172.21.15.104:0/1797746062 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc430005330 con 0x7fc4500b0860 2024-03-29T11:22:53.559 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.582+0000 7fc437fff640 1 -- 172.21.15.104:0/1797746062 <== mon.1 v1:172.21.15.119:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7fc4440520e0 con 0x7fc4500b0860 2024-03-29T11:22:53.559 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.582+0000 7fc437fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:53.559 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.582+0000 7fc437fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:53.559 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:22:53.560 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:53.564 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.586+0000 7fc455b89640 10 librados: watch_flush enter 2024-03-29T11:22:53.564 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.586+0000 7fc455b89640 10 librados: watch_flush exit 2024-03-29T11:22:53.564 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.586+0000 7fc455b89640 20 client.4333.objecter shutdown clearing up homeless session... 2024-03-29T11:22:53.564 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.586+0000 7fc455b89640 10 client.4333.objecter successfully canceled tick 2024-03-29T11:22:53.564 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.586+0000 7fc455b89640 1 -- 172.21.15.104:0/1797746062 >> v1:172.21.15.104:6816/3341126660 conn(0x7fc43c06dfe0 legacy=0x7fc43c0704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:53.564 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.586+0000 7fc455b89640 10 monclient: shutdown 2024-03-29T11:22:53.564 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.586+0000 7fc455b89640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:53.564 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.586+0000 7fc455b89640 1 -- 172.21.15.104:0/1797746062 >> v1:172.21.15.119:6789/0 conn(0x7fc4500b0860 legacy=0x7fc450136be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:53.564 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.586+0000 7fc455b89640 1 -- 172.21.15.104:0/1797746062 shutdown_connections 2024-03-29T11:22:53.564 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.586+0000 7fc455b89640 1 -- 172.21.15.104:0/1797746062 >> 172.21.15.104:0/1797746062 conn(0x7fc45006b660 msgr2=0x7fc4500ac900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:53.565 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.587+0000 7fc455b89640 1 -- 172.21.15.104:0/1797746062 shutdown_connections 2024-03-29T11:22:53.565 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.587+0000 7fc455b89640 1 -- 172.21.15.104:0/1797746062 wait complete. 2024-03-29T11:22:53.565 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:22:53.587+0000 7fc455b89640 1 librados: shutdown 2024-03-29T11:22:53.579 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:00.385 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:00.530 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.552+0000 7f44d58fb640 10 monclient: get_monmap_and_config 2024-03-29T11:23:00.530 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.552+0000 7f44d58fb640 10 monclient: build_initial_monmap 2024-03-29T11:23:00.530 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.552+0000 7f44d58fb640 10 monclient: monmap: 2024-03-29T11:23:00.530 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:23:00.531 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:23:00.531 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:23:00.553535+0000 2024-03-29T11:23:00.531 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:23:00.553535+0000 2024-03-29T11:23:00.531 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:00.531 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:00.531 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:23:00.531 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:23:00.531 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:23:00.531 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:00.533 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.556+0000 7f44d58fb640 1 Processor -- start 2024-03-29T11:23:00.534 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.556+0000 7f44d58fb640 1 -- start start 2024-03-29T11:23:00.534 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.556+0000 7f44d58fb640 10 monclient: init 2024-03-29T11:23:00.534 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.557+0000 7f44d58fb640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:00.535 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.557+0000 7f44d58fb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:23:00.538 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.558+0000 7f44d58fb640 10 monclient(hunting): picked mon.noname-b con 0x7f44d01457f0 addr v1:172.21.15.104:6790/0 2024-03-29T11:23:00.538 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.558+0000 7f44d58fb640 10 monclient(hunting): picked mon.noname-a con 0x7f44d013f650 addr v1:172.21.15.104:6789/0 2024-03-29T11:23:00.538 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.558+0000 7f44d58fb640 10 monclient(hunting): picked mon.noname-c con 0x7f44d00a9d70 addr v1:172.21.15.119:6789/0 2024-03-29T11:23:00.538 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.558+0000 7f44d58fb640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44d00aaef0 con 0x7f44d013f650 2024-03-29T11:23:00.538 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.558+0000 7f44d58fb640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44d0145bd0 con 0x7f44d00a9d70 2024-03-29T11:23:00.538 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.558+0000 7f44d58fb640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44d0145eb0 con 0x7f44d01457f0 2024-03-29T11:23:00.538 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.558+0000 7f44d58fb640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:00.538 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.558+0000 7f44d58fb640 10 monclient(hunting): authenticate will time out at 1161.809949s 2024-03-29T11:23:00.538 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.559+0000 7f44cf7fe640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f44d00a9d70 0x7f44d014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:38384/0 (socket says 172.21.15.104:38384) 2024-03-29T11:23:00.538 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.559+0000 7f44cf7fe640 1 -- 172.21.15.104:0/2325648599 learned_addr learned my addr 172.21.15.104:0/2325648599 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:23:00.539 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.560+0000 7f44cdffb640 1 -- 172.21.15.104:0/2325648599 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2808299730 0 0) 0x7f44d0145bd0 con 0x7f44d00a9d70 2024-03-29T11:23:00.539 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.560+0000 7f44cdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:00.539 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.560+0000 7f44cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:00.539 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.560+0000 7f44cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:00.539 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.560+0000 7f44cdffb640 10 monclient(hunting): my global_id is 4336 2024-03-29T11:23:00.539 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.560+0000 7f44cdffb640 1 -- 172.21.15.104:0/2325648599 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44b8003520 con 0x7f44d00a9d70 2024-03-29T11:23:00.539 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.560+0000 7f44cdffb640 1 -- 172.21.15.104:0/2325648599 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 693108299 0 0) 0x7f44d0145eb0 con 0x7f44d01457f0 2024-03-29T11:23:00.539 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.560+0000 7f44cdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:00.539 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.560+0000 7f44cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:00.539 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.560+0000 7f44cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:00.539 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.560+0000 7f44cdffb640 10 monclient(hunting): my global_id is 4373 2024-03-29T11:23:00.539 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.560+0000 7f44cdffb640 1 -- 172.21.15.104:0/2325648599 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44d0145bd0 con 0x7f44d01457f0 2024-03-29T11:23:00.539 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.560+0000 7f44cdffb640 1 -- 172.21.15.104:0/2325648599 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3771532974 0 0) 0x7f44b8003520 con 0x7f44d00a9d70 2024-03-29T11:23:00.539 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.560+0000 7f44cdffb640 1 -- 172.21.15.104:0/2325648599 >> v1:172.21.15.104:6790/0 conn(0x7f44d01457f0 legacy=0x7f44d013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:00.539 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.560+0000 7f44cdffb640 1 -- 172.21.15.104:0/2325648599 >> v1:172.21.15.104:6789/0 conn(0x7f44d013f650 legacy=0x7f44d00b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:23:00.542 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.563+0000 7f44cdffb640 10 monclient: _finish_hunting 0 2024-03-29T11:23:00.543 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.563+0000 7f44cdffb640 1 monclient: found mon.noname-c 2024-03-29T11:23:00.543 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.563+0000 7f44cdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:00.543 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.563+0000 7f44cdffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.119:6789/0 2024-03-29T11:23:00.543 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.563+0000 7f44cdffb640 1 -- 172.21.15.104:0/2325648599 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d0146340 con 0x7f44d00a9d70 2024-03-29T11:23:00.543 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.563+0000 7f44cdffb640 10 monclient: _finish_auth 0 2024-03-29T11:23:00.543 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.563+0000 7f44cdffb640 10 monclient: _check_auth_tickets 2024-03-29T11:23:00.543 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.563+0000 7f44cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:00.543 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.563+0000 7f44cdffb640 1 -- 172.21.15.104:0/2325648599 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f44c8001a10 con 0x7f44d00a9d70 2024-03-29T11:23:00.543 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.563+0000 7f44cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:00.543 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.563+0000 7f44cdffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:00.543 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.563+0000 7f44cdffb640 10 monclient: dump: 2024-03-29T11:23:00.543 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:23:00.543 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:23:00.543 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:00.544 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:00.544 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:00.544 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:00.544 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:23:00.544 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:23:00.544 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:23:00.544 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:00.544 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.563+0000 7f44cdffb640 1 -- 172.21.15.104:0/2325648599 <== mon.1 v1:172.21.15.119:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44c8001c30 con 0x7f44d00a9d70 2024-03-29T11:23:00.544 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.563+0000 7f44cdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:00.544 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.563+0000 7f44cdffb640 1 -- 172.21.15.104:0/2325648599 <== mon.1 v1:172.21.15.119:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f44c8001fc0 con 0x7f44d00a9d70 2024-03-29T11:23:00.544 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.563+0000 7f44cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:00.544 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.563+0000 7f44cdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:00.544 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.563+0000 7f44cdffb640 10 monclient: dump: 2024-03-29T11:23:00.544 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:23:00.544 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:23:00.545 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:00.545 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:00.545 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:00.545 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:00.545 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:23:00.545 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:23:00.545 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:23:00.545 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:00.545 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.563+0000 7f44d58fb640 5 monclient: authenticate success, global_id 4336 2024-03-29T11:23:00.545 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.563+0000 7f44d58fb640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:00.545 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.563+0000 7f44d58fb640 4 set_mon_vals no callback set 2024-03-29T11:23:00.545 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.564+0000 7f44d58fb640 10 monclient: shutdown 2024-03-29T11:23:00.545 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.564+0000 7f44d58fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:00.545 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.564+0000 7f44d58fb640 1 -- 172.21.15.104:0/2325648599 >> v1:172.21.15.119:6789/0 conn(0x7f44d00a9d70 legacy=0x7f44d014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:00.545 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.565+0000 7f44d58fb640 1 -- 172.21.15.104:0/2325648599 shutdown_connections 2024-03-29T11:23:00.545 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.565+0000 7f44d58fb640 1 -- 172.21.15.104:0/2325648599 >> 172.21.15.104:0/2325648599 conn(0x7f44d0066a00 msgr2=0x7f44d0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:00.546 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.565+0000 7f44d58fb640 1 -- 172.21.15.104:0/2325648599 shutdown_connections 2024-03-29T11:23:00.546 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.565+0000 7f44d58fb640 1 -- 172.21.15.104:0/2325648599 wait complete. 2024-03-29T11:23:00.547 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.566+0000 7f44d58fb640 10 monclient: build_initial_monmap 2024-03-29T11:23:00.547 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.566+0000 7f44d58fb640 10 monclient: monmap: 2024-03-29T11:23:00.547 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:23:00.547 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:00.547 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:23:00.548 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:23:00.548 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:00.548 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:00.548 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:23:00.548 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:23:00.548 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:23:00.548 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:00.548 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.566+0000 7f44d58fb640 1 librados: starting msgr at 2024-03-29T11:23:00.548 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.566+0000 7f44d58fb640 1 librados: starting objecter 2024-03-29T11:23:00.548 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.566+0000 7f44d58fb640 1 Processor -- start 2024-03-29T11:23:00.548 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.566+0000 7f44d58fb640 1 -- start start 2024-03-29T11:23:00.548 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.566+0000 7f44d58fb640 1 librados: setting wanted keys 2024-03-29T11:23:00.548 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.566+0000 7f44d58fb640 1 librados: calling monclient init 2024-03-29T11:23:00.548 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.566+0000 7f44d58fb640 10 monclient: init 2024-03-29T11:23:00.548 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.567+0000 7f44d58fb640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:00.549 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.567+0000 7f44d58fb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:23:00.549 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.569+0000 7f44d58fb640 10 monclient(hunting): picked mon.noname-a con 0x7f44d00a9d70 addr v1:172.21.15.104:6789/0 2024-03-29T11:23:00.549 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.569+0000 7f44d58fb640 10 monclient(hunting): picked mon.noname-c con 0x7f44d013f650 addr v1:172.21.15.104:6790/0 2024-03-29T11:23:00.549 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.569+0000 7f44d58fb640 10 monclient(hunting): picked mon.noname-b con 0x7f44d01457f0 addr v1:172.21.15.119:6789/0 2024-03-29T11:23:00.549 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.569+0000 7f44d58fb640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44d01dfb50 con 0x7f44d00a9d70 2024-03-29T11:23:00.549 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.569+0000 7f44d58fb640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44d0070ea0 con 0x7f44d01457f0 2024-03-29T11:23:00.549 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.569+0000 7f44d58fb640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44d0071180 con 0x7f44d013f650 2024-03-29T11:23:00.549 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.569+0000 7f44d58fb640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:00.549 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.569+0000 7f44d58fb640 10 monclient(hunting): authenticate will time out at 1161.820184s 2024-03-29T11:23:00.552 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.571+0000 7f44ceffd640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f44d00a9d70 0x7f44d01ded30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:54499/0 (socket says 172.21.15.104:54499) 2024-03-29T11:23:00.552 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.571+0000 7f44ceffd640 1 -- 172.21.15.104:0/3912651671 learned_addr learned my addr 172.21.15.104:0/3912651671 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:23:00.552 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.571+0000 7f44b7fff640 10 client.?.objecter ms_handle_connect 0x7f44d01457f0 2024-03-29T11:23:00.552 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.571+0000 7f44b7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:00.552 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.571+0000 7f44b7fff640 10 client.?.objecter ms_handle_connect 0x7f44d00a9d70 2024-03-29T11:23:00.552 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.571+0000 7f44b7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:00.552 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.571+0000 7f44b7fff640 1 -- 172.21.15.104:0/3912651671 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1418577985 0 0) 0x7f44d01dfb50 con 0x7f44d00a9d70 2024-03-29T11:23:00.553 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.571+0000 7f44b7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:00.553 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.571+0000 7f44b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:00.553 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.571+0000 7f44b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:00.553 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.571+0000 7f44b7fff640 10 monclient(hunting): my global_id is 4362 2024-03-29T11:23:00.553 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.571+0000 7f44b7fff640 1 -- 172.21.15.104:0/3912651671 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44bc004eb0 con 0x7f44d00a9d70 2024-03-29T11:23:00.553 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.572+0000 7f44b7fff640 1 -- 172.21.15.104:0/3912651671 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2322997418 0 0) 0x7f44bc004eb0 con 0x7f44d00a9d70 2024-03-29T11:23:00.553 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.572+0000 7f44b7fff640 1 -- 172.21.15.104:0/3912651671 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f44d01dfb50 con 0x7f44d00a9d70 2024-03-29T11:23:00.554 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.572+0000 7f44b7fff640 1 -- 172.21.15.104:0/3912651671 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 830100977 0 0) 0x7f44d0070ea0 con 0x7f44d01457f0 2024-03-29T11:23:00.554 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.572+0000 7f44b7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:00.554 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.572+0000 7f44b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:00.554 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.572+0000 7f44b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:00.554 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.572+0000 7f44b7fff640 10 monclient(hunting): my global_id is 4339 2024-03-29T11:23:00.554 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.572+0000 7f44b7fff640 1 -- 172.21.15.104:0/3912651671 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44bc004eb0 con 0x7f44d01457f0 2024-03-29T11:23:00.554 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.572+0000 7f44b7fff640 1 -- 172.21.15.104:0/3912651671 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f44c0001890 con 0x7f44d00a9d70 2024-03-29T11:23:00.554 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.572+0000 7f44b7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:00.554 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.572+0000 7f44b7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:00.554 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.572+0000 7f44b7fff640 10 monclient(hunting): dump: 2024-03-29T11:23:00.554 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:23:00.554 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:23:00.554 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:00.554 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:00.555 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:00.555 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:00.555 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:23:00.555 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:23:00.555 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:23:00.555 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:00.555 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.572+0000 7f44b7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:00.555 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.572+0000 7f44b7fff640 1 -- 172.21.15.104:0/3912651671 <== mon.0 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3148088125 0 0) 0x7f44d01dfb50 con 0x7f44d00a9d70 2024-03-29T11:23:00.555 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.572+0000 7f44b7fff640 1 -- 172.21.15.104:0/3912651671 >> v1:172.21.15.104:6790/0 conn(0x7f44d013f650 legacy=0x7f44d01df440 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:23:00.555 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.573+0000 7f44b7fff640 1 -- 172.21.15.104:0/3912651671 >> v1:172.21.15.119:6789/0 conn(0x7f44d01457f0 legacy=0x7f44d01e7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:00.555 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.573+0000 7f44b7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:23:00.555 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.573+0000 7f44b7fff640 1 monclient: found mon.a 2024-03-29T11:23:00.555 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.573+0000 7f44b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:00.555 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.573+0000 7f44b7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:23:00.555 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.573+0000 7f44b7fff640 1 -- 172.21.15.104:0/3912651671 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d01dbdc0 con 0x7f44d00a9d70 2024-03-29T11:23:00.556 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.574+0000 7f44b7fff640 10 monclient: _finish_auth 0 2024-03-29T11:23:00.557 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.574+0000 7f44b7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:23:00.557 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.574+0000 7f44b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:00.558 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.574+0000 7f44d58fb640 5 monclient: authenticate success, global_id 4362 2024-03-29T11:23:00.558 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.574+0000 7f44d58fb640 10 monclient: _renew_subs 2024-03-29T11:23:00.558 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.574+0000 7f44d58fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:23:00.558 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.574+0000 7f44d58fb640 1 -- 172.21.15.104:0/3912651671 --> v1:172.21.15.104:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44d00b1690 con 0x7f44d00a9d70 2024-03-29T11:23:00.558 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.574+0000 7f44d58fb640 10 client.4362.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:00.558 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.574+0000 7f44d58fb640 10 monclient: _renew_subs 2024-03-29T11:23:00.558 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.574+0000 7f44d58fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:23:00.558 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.574+0000 7f44d58fb640 1 -- 172.21.15.104:0/3912651671 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f44d01dba10 con 0x7f44d00a9d70 2024-03-29T11:23:00.558 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.574+0000 7f44d58fb640 1 librados: init done 2024-03-29T11:23:00.558 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.575+0000 7f44b7fff640 1 -- 172.21.15.104:0/3912651671 <== mon.0 v1:172.21.15.104:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44c0001a60 con 0x7f44d00a9d70 2024-03-29T11:23:00.558 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.575+0000 7f44b7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:00.559 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.575+0000 7f44b7fff640 1 -- 172.21.15.104:0/3912651671 <== mon.0 v1:172.21.15.104:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f44c00037b0 con 0x7f44d00a9d70 2024-03-29T11:23:00.559 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.575+0000 7f44b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:00.559 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.575+0000 7f44b7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:00.559 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.575+0000 7f44b7fff640 10 monclient: dump: 2024-03-29T11:23:00.559 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:23:00.559 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:23:00.559 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:00.559 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:00.559 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:00.559 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:00.559 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:23:00.559 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:23:00.559 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:23:00.560 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:00.560 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.576+0000 7f44b5ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:00.560 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.576+0000 7f44b5ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:00.560 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.576+0000 7f44b5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:23:00.560 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.576+0000 7f44b5ffb640 1 -- 172.21.15.104:0/3912651671 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44d0075f10 con 0x7f44d00a9d70 2024-03-29T11:23:00.560 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.576+0000 7f44cd7fa640 4 set_mon_vals no callback set 2024-03-29T11:23:00.560 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.577+0000 7f44b7fff640 1 -- 172.21.15.104:0/3912651671 <== mon.0 v1:172.21.15.104:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f44c0003910 con 0x7f44d00a9d70 2024-03-29T11:23:00.560 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.579+0000 7f44b7fff640 1 -- 172.21.15.104:0/3912651671 <== mon.0 v1:172.21.15.104:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f44c0084970 con 0x7f44d00a9d70 2024-03-29T11:23:00.560 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.579+0000 7f44b7fff640 10 client.4362.objecter ms_dispatch 0x7f44d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:00.560 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.579+0000 7f44b7fff640 3 client.4362.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:00.560 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.579+0000 7f44b7fff640 3 client.4362.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:00.560 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.579+0000 7f44ce7fc640 10 monclient: get_auth_request con 0x7f44bc06dc90 auth_method 0 2024-03-29T11:23:00.560 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.579+0000 7f44b7fff640 20 client.4362.objecter dump_active .. 0 homeless 2024-03-29T11:23:00.561 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.579+0000 7f44b7fff640 10 client.4362.objecter ms_handle_connect 0x7f44bc06dc90 2024-03-29T11:23:00.563 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.586+0000 7f44b7fff640 1 -- 172.21.15.104:0/3912651671 <== mon.0 v1:172.21.15.104:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f44c0050080 con 0x7f44d00a9d70 2024-03-29T11:23:00.563 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.586+0000 7f44b7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:00.563 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.586+0000 7f44b7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:00.891 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.913+0000 7f44b5ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:00.891 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.913+0000 7f44b5ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:00.891 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.913+0000 7f44b5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:23:00.891 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.913+0000 7f44b5ffb640 1 -- 172.21.15.104:0/3912651671 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f44d01dbf50 con 0x7f44d00a9d70 2024-03-29T11:23:00.892 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.914+0000 7f44b7fff640 1 -- 172.21.15.104:0/3912651671 <== mon.0 v1:172.21.15.104:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f44c0052180 con 0x7f44d00a9d70 2024-03-29T11:23:00.892 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.914+0000 7f44b7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:00.892 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.914+0000 7f44b7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:00.895 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:23:00.895 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:00.898 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.921+0000 7f44b5ffb640 10 librados: watch_flush enter 2024-03-29T11:23:00.900 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.922+0000 7f44b5ffb640 10 librados: watch_flush exit 2024-03-29T11:23:00.900 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.922+0000 7f44b5ffb640 20 client.4362.objecter shutdown clearing up homeless session... 2024-03-29T11:23:00.900 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.922+0000 7f44b5ffb640 10 client.4362.objecter successfully canceled tick 2024-03-29T11:23:00.901 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.923+0000 7f44b5ffb640 1 -- 172.21.15.104:0/3912651671 >> v1:172.21.15.104:6816/3341126660 conn(0x7f44bc06dc90 legacy=0x7f44bc070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:00.901 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.923+0000 7f44b5ffb640 10 monclient: shutdown 2024-03-29T11:23:00.901 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.923+0000 7f44b5ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:00.901 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.923+0000 7f44b5ffb640 1 -- 172.21.15.104:0/3912651671 >> v1:172.21.15.104:6789/0 conn(0x7f44d00a9d70 legacy=0x7f44d01ded30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:00.901 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.923+0000 7f44b5ffb640 1 -- 172.21.15.104:0/3912651671 shutdown_connections 2024-03-29T11:23:00.901 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.923+0000 7f44b5ffb640 1 -- 172.21.15.104:0/3912651671 >> 172.21.15.104:0/3912651671 conn(0x7f44d0066a00 msgr2=0x7f44d00b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:00.902 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.923+0000 7f44b5ffb640 1 -- 172.21.15.104:0/3912651671 shutdown_connections 2024-03-29T11:23:00.902 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.923+0000 7f44b5ffb640 1 -- 172.21.15.104:0/3912651671 wait complete. 2024-03-29T11:23:00.902 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:00.924+0000 7f44b5ffb640 1 librados: shutdown 2024-03-29T11:23:00.918 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:07.725 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:07.878 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.899+0000 7fa3ce14d640 10 monclient: get_monmap_and_config 2024-03-29T11:23:07.878 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.899+0000 7fa3ce14d640 10 monclient: build_initial_monmap 2024-03-29T11:23:07.878 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.899+0000 7fa3ce14d640 10 monclient: monmap: 2024-03-29T11:23:07.878 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:23:07.878 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:23:07.878 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:23:07.900525+0000 2024-03-29T11:23:07.878 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:23:07.900525+0000 2024-03-29T11:23:07.878 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:07.878 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:07.879 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:23:07.879 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:23:07.879 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:23:07.879 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:07.879 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.901+0000 7fa3ce14d640 1 Processor -- start 2024-03-29T11:23:07.879 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.901+0000 7fa3ce14d640 1 -- start start 2024-03-29T11:23:07.880 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.901+0000 7fa3ce14d640 10 monclient: init 2024-03-29T11:23:07.880 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.902+0000 7fa3ce14d640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:07.880 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.902+0000 7fa3ce14d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:23:07.883 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.906+0000 7fa3ce14d640 10 monclient(hunting): picked mon.noname-a con 0x7fa3c80aac30 addr v1:172.21.15.104:6789/0 2024-03-29T11:23:07.884 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.906+0000 7fa3ce14d640 10 monclient(hunting): picked mon.noname-c con 0x7fa3c8072f20 addr v1:172.21.15.119:6789/0 2024-03-29T11:23:07.887 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.909+0000 7fa3cd14b640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fa3c80aac30 0x7fa3c813af50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:41591/0 (socket says 172.21.15.104:41591) 2024-03-29T11:23:07.888 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.909+0000 7fa3cd94c640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7fa3c80a9e90 0x7fa3c80ad1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:47355/0 (socket says 172.21.15.104:47355) 2024-03-29T11:23:07.888 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.909+0000 7fa3cd94c640 1 -- 172.21.15.104:0/3728209852 learned_addr learned my addr 172.21.15.104:0/3728209852 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:23:07.888 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.909+0000 7fa3ce14d640 10 monclient(hunting): picked mon.noname-b con 0x7fa3c80a9e90 addr v1:172.21.15.104:6790/0 2024-03-29T11:23:07.888 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.909+0000 7fa3ce14d640 1 -- 172.21.15.104:0/3728209852 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3c80ad8e0 con 0x7fa3c80aac30 2024-03-29T11:23:07.888 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.909+0000 7fa3ce14d640 1 -- 172.21.15.104:0/3728209852 --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3c80ac6a0 con 0x7fa3c8072f20 2024-03-29T11:23:07.888 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.909+0000 7fa3ce14d640 1 -- 172.21.15.104:0/3728209852 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3c80ab230 con 0x7fa3c80a9e90 2024-03-29T11:23:07.888 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.909+0000 7fa3bffff640 1 -- 172.21.15.104:0/3728209852 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1910978654 0 0) 0x7fa3c80ad8e0 con 0x7fa3c80aac30 2024-03-29T11:23:07.888 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.910+0000 7fa3ce14d640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:07.888 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.910+0000 7fa3ce14d640 10 monclient(hunting): authenticate will time out at 1169.161105s 2024-03-29T11:23:07.890 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.911+0000 7fa3bffff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:07.891 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.911+0000 7fa3bffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:07.891 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.911+0000 7fa3bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:07.891 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.911+0000 7fa3bffff640 10 monclient(hunting): my global_id is 4365 2024-03-29T11:23:07.891 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.911+0000 7fa3bffff640 1 -- 172.21.15.104:0/3728209852 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3b4003520 con 0x7fa3c80aac30 2024-03-29T11:23:07.891 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.911+0000 7fa3bffff640 1 -- 172.21.15.104:0/3728209852 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3746522935 0 0) 0x7fa3c80ab230 con 0x7fa3c80a9e90 2024-03-29T11:23:07.891 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.911+0000 7fa3bffff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:07.891 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.911+0000 7fa3bffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:07.891 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.911+0000 7fa3bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:07.891 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.911+0000 7fa3bffff640 10 monclient(hunting): my global_id is 4376 2024-03-29T11:23:07.891 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.911+0000 7fa3bffff640 1 -- 172.21.15.104:0/3728209852 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3c80ad8e0 con 0x7fa3c80a9e90 2024-03-29T11:23:07.892 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.913+0000 7fa3bffff640 1 -- 172.21.15.104:0/3728209852 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2329717972 0 0) 0x7fa3c80ad8e0 con 0x7fa3c80a9e90 2024-03-29T11:23:07.892 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.913+0000 7fa3bffff640 1 -- 172.21.15.104:0/3728209852 >> v1:172.21.15.119:6789/0 conn(0x7fa3c8072f20 legacy=0x7fa3c80acac0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:23:07.892 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.913+0000 7fa3bffff640 1 -- 172.21.15.104:0/3728209852 >> v1:172.21.15.104:6789/0 conn(0x7fa3c80aac30 legacy=0x7fa3c813af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:07.892 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.913+0000 7fa3bffff640 10 monclient: _finish_hunting 0 2024-03-29T11:23:07.892 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.913+0000 7fa3bffff640 1 monclient: found mon.noname-b 2024-03-29T11:23:07.892 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.913+0000 7fa3bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:07.892 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.913+0000 7fa3bffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.104:6790/0 2024-03-29T11:23:07.892 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.913+0000 7fa3bffff640 1 -- 172.21.15.104:0/3728209852 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c813b660 con 0x7fa3c80a9e90 2024-03-29T11:23:07.892 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.913+0000 7fa3bffff640 10 monclient: _finish_auth 0 2024-03-29T11:23:07.892 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.913+0000 7fa3bffff640 10 monclient: _check_auth_tickets 2024-03-29T11:23:07.893 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.913+0000 7fa3bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:07.893 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.913+0000 7fa3bffff640 1 -- 172.21.15.104:0/3728209852 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fa3c00019f0 con 0x7fa3c80a9e90 2024-03-29T11:23:07.893 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.913+0000 7fa3bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:07.893 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.913+0000 7fa3bffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:07.893 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.913+0000 7fa3bffff640 10 monclient: dump: 2024-03-29T11:23:07.893 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:23:07.893 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:23:07.893 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:07.893 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:07.893 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:07.893 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:07.893 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:23:07.893 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:23:07.893 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:23:07.893 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:07.893 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.913+0000 7fa3bffff640 1 -- 172.21.15.104:0/3728209852 <== mon.2 v1:172.21.15.104:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa3c0001c10 con 0x7fa3c80a9e90 2024-03-29T11:23:07.894 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.914+0000 7fa3bffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:07.894 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.914+0000 7fa3bffff640 1 -- 172.21.15.104:0/3728209852 <== mon.2 v1:172.21.15.104:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fa3c0001fa0 con 0x7fa3c80a9e90 2024-03-29T11:23:07.894 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.914+0000 7fa3bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:07.894 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.914+0000 7fa3bffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:07.894 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.914+0000 7fa3bffff640 10 monclient: dump: 2024-03-29T11:23:07.894 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:23:07.894 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:23:07.894 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:07.894 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:07.894 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:07.894 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:07.894 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:23:07.894 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:23:07.894 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:23:07.894 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:07.895 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.914+0000 7fa3ce14d640 5 monclient: authenticate success, global_id 4376 2024-03-29T11:23:07.895 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.914+0000 7fa3ce14d640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:07.895 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.914+0000 7fa3ce14d640 4 set_mon_vals no callback set 2024-03-29T11:23:07.895 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.915+0000 7fa3ce14d640 10 monclient: shutdown 2024-03-29T11:23:07.895 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.915+0000 7fa3ce14d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:07.895 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.915+0000 7fa3ce14d640 1 -- 172.21.15.104:0/3728209852 >> v1:172.21.15.104:6790/0 conn(0x7fa3c80a9e90 legacy=0x7fa3c80ad1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:07.895 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.915+0000 7fa3ce14d640 1 -- 172.21.15.104:0/3728209852 shutdown_connections 2024-03-29T11:23:07.895 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.915+0000 7fa3ce14d640 1 -- 172.21.15.104:0/3728209852 >> 172.21.15.104:0/3728209852 conn(0x7fa3c80b3a30 msgr2=0x7fa3c80b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:07.897 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.919+0000 7fa3ce14d640 1 -- 172.21.15.104:0/3728209852 shutdown_connections 2024-03-29T11:23:07.897 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.919+0000 7fa3ce14d640 1 -- 172.21.15.104:0/3728209852 wait complete. 2024-03-29T11:23:07.899 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.920+0000 7fa3ce14d640 10 monclient: build_initial_monmap 2024-03-29T11:23:07.899 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.920+0000 7fa3ce14d640 10 monclient: monmap: 2024-03-29T11:23:07.899 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:23:07.900 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:07.900 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:23:07.900 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:23:07.900 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:07.900 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:07.900 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:23:07.900 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:23:07.900 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:23:07.900 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:07.900 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.921+0000 7fa3ce14d640 1 librados: starting msgr at 2024-03-29T11:23:07.900 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.921+0000 7fa3ce14d640 1 librados: starting objecter 2024-03-29T11:23:07.900 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.921+0000 7fa3ce14d640 1 Processor -- start 2024-03-29T11:23:07.900 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.921+0000 7fa3ce14d640 1 -- start start 2024-03-29T11:23:07.900 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.921+0000 7fa3ce14d640 1 librados: setting wanted keys 2024-03-29T11:23:07.900 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.921+0000 7fa3ce14d640 1 librados: calling monclient init 2024-03-29T11:23:07.901 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.922+0000 7fa3ce14d640 10 monclient: init 2024-03-29T11:23:07.901 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.922+0000 7fa3ce14d640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:07.901 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.922+0000 7fa3ce14d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:23:07.902 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.924+0000 7fa3ce14d640 10 monclient(hunting): picked mon.noname-c con 0x7fa3c8072f20 addr v1:172.21.15.104:6790/0 2024-03-29T11:23:07.904 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.924+0000 7fa3ce14d640 10 monclient(hunting): picked mon.noname-b con 0x7fa3c80a9e90 addr v1:172.21.15.119:6789/0 2024-03-29T11:23:07.904 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.924+0000 7fa3ce14d640 10 monclient(hunting): picked mon.noname-a con 0x7fa3c80aac30 addr v1:172.21.15.104:6789/0 2024-03-29T11:23:07.904 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.924+0000 7fa3ce14d640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3c8132880 con 0x7fa3c80aac30 2024-03-29T11:23:07.904 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.924+0000 7fa3ce14d640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3c8072800 con 0x7fa3c80a9e90 2024-03-29T11:23:07.904 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.924+0000 7fa3ce14d640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3c806f360 con 0x7fa3c8072f20 2024-03-29T11:23:07.904 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.924+0000 7fa3ce14d640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:07.904 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.924+0000 7fa3ce14d640 10 monclient(hunting): authenticate will time out at 1169.175869s 2024-03-29T11:23:07.904 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.925+0000 7fa3cd14b640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7fa3c8072f20 0x7fa3c81d6020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:47481/0 (socket says 172.21.15.104:47481) 2024-03-29T11:23:07.904 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.925+0000 7fa3cd14b640 1 -- 172.21.15.104:0/877753577 learned_addr learned my addr 172.21.15.104:0/877753577 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:23:07.906 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.928+0000 7fa3bdffb640 10 client.?.objecter ms_handle_connect 0x7fa3c8072f20 2024-03-29T11:23:07.906 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.928+0000 7fa3bdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:07.911 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.929+0000 7fa3bdffb640 10 client.?.objecter ms_handle_connect 0x7fa3c80aac30 2024-03-29T11:23:07.911 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.929+0000 7fa3bdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:07.911 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.929+0000 7fa3bdffb640 1 -- 172.21.15.104:0/877753577 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3033580568 0 0) 0x7fa3c806f360 con 0x7fa3c8072f20 2024-03-29T11:23:07.911 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.929+0000 7fa3bdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:07.912 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.929+0000 7fa3bdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:07.912 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.929+0000 7fa3bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:07.912 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.929+0000 7fa3bdffb640 10 monclient(hunting): my global_id is 4379 2024-03-29T11:23:07.912 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.929+0000 7fa3bdffb640 1 -- 172.21.15.104:0/877753577 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3ac004eb0 con 0x7fa3c8072f20 2024-03-29T11:23:07.912 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.929+0000 7fa3bdffb640 1 -- 172.21.15.104:0/877753577 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2998111580 0 0) 0x7fa3ac004eb0 con 0x7fa3c8072f20 2024-03-29T11:23:07.912 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.929+0000 7fa3bdffb640 1 -- 172.21.15.104:0/877753577 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3c806f360 con 0x7fa3c8072f20 2024-03-29T11:23:07.912 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.929+0000 7fa3bdffb640 1 -- 172.21.15.104:0/877753577 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3475558356 0 0) 0x7fa3c8132880 con 0x7fa3c80aac30 2024-03-29T11:23:07.912 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.929+0000 7fa3bdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:07.912 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.929+0000 7fa3bdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:07.912 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.929+0000 7fa3bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:07.912 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.929+0000 7fa3bdffb640 10 monclient(hunting): my global_id is 4368 2024-03-29T11:23:07.912 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.929+0000 7fa3bdffb640 1 -- 172.21.15.104:0/877753577 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3ac005590 con 0x7fa3c80aac30 2024-03-29T11:23:07.912 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.930+0000 7fa3bdffb640 1 -- 172.21.15.104:0/877753577 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fa3c40018c0 con 0x7fa3c8072f20 2024-03-29T11:23:07.912 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.930+0000 7fa3bdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:07.913 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.930+0000 7fa3bdffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:07.913 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.930+0000 7fa3bdffb640 10 monclient(hunting): dump: 2024-03-29T11:23:07.913 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:23:07.913 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:23:07.913 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:07.913 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:07.913 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:07.913 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:07.913 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:23:07.913 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:23:07.913 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:23:07.913 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:07.913 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.930+0000 7fa3bdffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:07.913 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.930+0000 7fa3bdffb640 10 client.?.objecter ms_handle_connect 0x7fa3c80a9e90 2024-03-29T11:23:07.914 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.930+0000 7fa3bdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:07.914 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.930+0000 7fa3bdffb640 1 -- 172.21.15.104:0/877753577 <== mon.2 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2190197893 0 0) 0x7fa3c806f360 con 0x7fa3c8072f20 2024-03-29T11:23:07.914 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.930+0000 7fa3bdffb640 1 -- 172.21.15.104:0/877753577 >> v1:172.21.15.119:6789/0 conn(0x7fa3c80a9e90 legacy=0x7fa3c81d6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:07.914 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.930+0000 7fa3bdffb640 1 -- 172.21.15.104:0/877753577 >> v1:172.21.15.104:6789/0 conn(0x7fa3c80aac30 legacy=0x7fa3c81dd2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:07.914 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.930+0000 7fa3bdffb640 10 monclient: _finish_hunting 0 2024-03-29T11:23:07.914 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.930+0000 7fa3bdffb640 1 monclient: found mon.c 2024-03-29T11:23:07.914 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.930+0000 7fa3bdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:07.914 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.930+0000 7fa3bdffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:23:07.914 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.930+0000 7fa3bdffb640 1 -- 172.21.15.104:0/877753577 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c806f4f0 con 0x7fa3c8072f20 2024-03-29T11:23:07.914 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.930+0000 7fa3bdffb640 10 monclient: _finish_auth 0 2024-03-29T11:23:07.914 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.930+0000 7fa3bdffb640 10 monclient: _check_auth_tickets 2024-03-29T11:23:07.914 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.930+0000 7fa3bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:07.914 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.930+0000 7fa3ce14d640 5 monclient: authenticate success, global_id 4379 2024-03-29T11:23:07.915 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.931+0000 7fa3ce14d640 10 monclient: _renew_subs 2024-03-29T11:23:07.915 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.931+0000 7fa3ce14d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:23:07.915 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.931+0000 7fa3ce14d640 1 -- 172.21.15.104:0/877753577 --> v1:172.21.15.104:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3c8071c30 con 0x7fa3c8072f20 2024-03-29T11:23:07.915 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.931+0000 7fa3ce14d640 10 client.4379.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:07.915 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.931+0000 7fa3ce14d640 10 monclient: _renew_subs 2024-03-29T11:23:07.915 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.931+0000 7fa3ce14d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:23:07.915 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.931+0000 7fa3ce14d640 1 -- 172.21.15.104:0/877753577 --> v1:172.21.15.104:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3c8131950 con 0x7fa3c8072f20 2024-03-29T11:23:07.915 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.931+0000 7fa3ce14d640 1 librados: init done 2024-03-29T11:23:07.915 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.933+0000 7fa3ce14d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:07.915 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.933+0000 7fa3ce14d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:07.915 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.933+0000 7fa3ce14d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:23:07.915 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.933+0000 7fa3ce14d640 1 -- 172.21.15.104:0/877753577 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3c8072990 con 0x7fa3c8072f20 2024-03-29T11:23:07.915 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.933+0000 7fa3bdffb640 1 -- 172.21.15.104:0/877753577 <== mon.2 v1:172.21.15.104:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa3c40047e0 con 0x7fa3c8072f20 2024-03-29T11:23:07.915 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.933+0000 7fa3bdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:07.915 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.933+0000 7fa3beffd640 4 set_mon_vals no callback set 2024-03-29T11:23:07.917 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.934+0000 7fa3bdffb640 1 -- 172.21.15.104:0/877753577 <== mon.2 v1:172.21.15.104:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fa3c40018c0 con 0x7fa3c8072f20 2024-03-29T11:23:07.917 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.934+0000 7fa3bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:07.917 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.934+0000 7fa3bdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:07.917 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.934+0000 7fa3bdffb640 10 monclient: dump: 2024-03-29T11:23:07.917 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:23:07.917 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:23:07.917 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:07.917 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:07.917 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:07.917 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:07.917 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:23:07.917 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:23:07.917 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:23:07.917 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:07.917 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.935+0000 7fa3bdffb640 1 -- 172.21.15.104:0/877753577 <== mon.2 v1:172.21.15.104:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7fa3c4004a50 con 0x7fa3c8072f20 2024-03-29T11:23:07.918 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.936+0000 7fa3bdffb640 1 -- 172.21.15.104:0/877753577 <== mon.2 v1:172.21.15.104:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7fa3c4054510 con 0x7fa3c8072f20 2024-03-29T11:23:07.918 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.936+0000 7fa3bdffb640 10 client.4379.objecter ms_dispatch 0x7fa3c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:07.918 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.936+0000 7fa3bdffb640 3 client.4379.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:07.918 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.936+0000 7fa3bdffb640 3 client.4379.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:07.918 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.936+0000 7fa3bdffb640 20 client.4379.objecter dump_active .. 0 homeless 2024-03-29T11:23:07.918 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.936+0000 7fa3cc94a640 10 monclient: get_auth_request con 0x7fa3ac06e220 auth_method 0 2024-03-29T11:23:07.918 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.937+0000 7fa3bdffb640 10 client.4379.objecter ms_handle_connect 0x7fa3ac06e220 2024-03-29T11:23:07.925 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.947+0000 7fa3bdffb640 1 -- 172.21.15.104:0/877753577 <== mon.2 v1:172.21.15.104:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa3c40507b0 con 0x7fa3c8072f20 2024-03-29T11:23:07.925 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.947+0000 7fa3bdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:07.925 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:07.947+0000 7fa3bdffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:08.255 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:08.277+0000 7fa3ce14d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:08.255 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:08.277+0000 7fa3ce14d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:08.255 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:08.277+0000 7fa3ce14d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:23:08.255 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:08.277+0000 7fa3ce14d640 1 -- 172.21.15.104:0/877753577 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa3c804f730 con 0x7fa3c8072f20 2024-03-29T11:23:08.256 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:08.277+0000 7fa3bdffb640 1 -- 172.21.15.104:0/877753577 <== mon.2 v1:172.21.15.104:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7fa3c4050950 con 0x7fa3c8072f20 2024-03-29T11:23:08.256 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:08.277+0000 7fa3bdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:08.256 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:08.277+0000 7fa3bdffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:08.256 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:23:08.256 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:08.260 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:08.282+0000 7fa3ce14d640 10 librados: watch_flush enter 2024-03-29T11:23:08.260 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:08.282+0000 7fa3ce14d640 10 librados: watch_flush exit 2024-03-29T11:23:08.260 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:08.282+0000 7fa3ce14d640 20 client.4379.objecter shutdown clearing up homeless session... 2024-03-29T11:23:08.261 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:08.282+0000 7fa3ce14d640 10 client.4379.objecter successfully canceled tick 2024-03-29T11:23:08.261 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:08.282+0000 7fa3ce14d640 1 -- 172.21.15.104:0/877753577 >> v1:172.21.15.104:6816/3341126660 conn(0x7fa3ac06e220 legacy=0x7fa3ac0706e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:08.261 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:08.282+0000 7fa3ce14d640 10 monclient: shutdown 2024-03-29T11:23:08.261 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:08.282+0000 7fa3ce14d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:08.261 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:08.282+0000 7fa3ce14d640 1 -- 172.21.15.104:0/877753577 >> v1:172.21.15.104:6790/0 conn(0x7fa3c8072f20 legacy=0x7fa3c81d6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:08.261 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:08.282+0000 7fa3ce14d640 1 -- 172.21.15.104:0/877753577 shutdown_connections 2024-03-29T11:23:08.261 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:08.282+0000 7fa3ce14d640 1 -- 172.21.15.104:0/877753577 >> 172.21.15.104:0/877753577 conn(0x7fa3c80b3a30 msgr2=0x7fa3c80aef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:08.261 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:08.282+0000 7fa3ce14d640 1 -- 172.21.15.104:0/877753577 shutdown_connections 2024-03-29T11:23:08.261 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:08.283+0000 7fa3ce14d640 1 -- 172.21.15.104:0/877753577 wait complete. 2024-03-29T11:23:08.261 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:08.283+0000 7fa3ce14d640 1 librados: shutdown 2024-03-29T11:23:08.278 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:15.084 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:15.239 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.259+0000 7efe78264640 10 monclient: get_monmap_and_config 2024-03-29T11:23:15.239 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.259+0000 7efe78264640 10 monclient: build_initial_monmap 2024-03-29T11:23:15.239 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.259+0000 7efe78264640 10 monclient: monmap: 2024-03-29T11:23:15.239 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:23:15.239 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:23:15.239 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:23:15.260333+0000 2024-03-29T11:23:15.239 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:23:15.260333+0000 2024-03-29T11:23:15.239 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:15.239 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:15.239 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:23:15.239 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:23:15.239 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:23:15.240 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:15.240 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.262+0000 7efe78264640 1 Processor -- start 2024-03-29T11:23:15.243 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.262+0000 7efe78264640 1 -- start start 2024-03-29T11:23:15.243 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.262+0000 7efe78264640 10 monclient: init 2024-03-29T11:23:15.244 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.263+0000 7efe78264640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:15.244 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.263+0000 7efe78264640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:23:15.244 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.263+0000 7efe78264640 10 monclient(hunting): picked mon.noname-c con 0x7efe700b0770 addr v1:172.21.15.119:6789/0 2024-03-29T11:23:15.244 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.263+0000 7efe78264640 10 monclient(hunting): picked mon.noname-b con 0x7efe7014a700 addr v1:172.21.15.104:6790/0 2024-03-29T11:23:15.244 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.263+0000 7efe78264640 10 monclient(hunting): picked mon.noname-a con 0x7efe700ac5a0 addr v1:172.21.15.104:6789/0 2024-03-29T11:23:15.244 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.263+0000 7efe78264640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe700aba70 con 0x7efe700ac5a0 2024-03-29T11:23:15.244 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.263+0000 7efe78264640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe700aecb0 con 0x7efe700b0770 2024-03-29T11:23:15.244 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.263+0000 7efe78264640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe700af150 con 0x7efe7014a700 2024-03-29T11:23:15.244 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.263+0000 7efe78264640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:15.244 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.263+0000 7efe78264640 10 monclient(hunting): authenticate will time out at 1176.514629s 2024-03-29T11:23:15.249 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.268+0000 7efe76000640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7efe700b0770 0x7efe70149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:50023/0 (socket says 172.21.15.104:50023) 2024-03-29T11:23:15.249 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.268+0000 7efe76000640 1 -- 172.21.15.104:0/1140335454 learned_addr learned my addr 172.21.15.104:0/1140335454 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:23:15.249 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.268+0000 7efe74ffe640 1 -- 172.21.15.104:0/1140335454 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4100543527 0 0) 0x7efe700aecb0 con 0x7efe700b0770 2024-03-29T11:23:15.249 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.268+0000 7efe74ffe640 10 monclient(hunting): _negotiate 2024-03-29T11:23:15.249 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.268+0000 7efe74ffe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:15.249 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.268+0000 7efe74ffe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:15.249 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.268+0000 7efe74ffe640 10 monclient(hunting): my global_id is 4342 2024-03-29T11:23:15.249 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.268+0000 7efe74ffe640 1 -- 172.21.15.104:0/1140335454 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe58003520 con 0x7efe700b0770 2024-03-29T11:23:15.249 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.269+0000 7efe74ffe640 1 -- 172.21.15.104:0/1140335454 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 534325715 0 0) 0x7efe58003520 con 0x7efe700b0770 2024-03-29T11:23:15.249 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.269+0000 7efe74ffe640 1 -- 172.21.15.104:0/1140335454 >> v1:172.21.15.104:6790/0 conn(0x7efe7014a700 legacy=0x7efe700abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:23:15.249 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.270+0000 7efe74ffe640 1 -- 172.21.15.104:0/1140335454 >> v1:172.21.15.104:6789/0 conn(0x7efe700ac5a0 legacy=0x7efe70151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:23:15.249 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.270+0000 7efe74ffe640 10 monclient: _finish_hunting 0 2024-03-29T11:23:15.249 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.270+0000 7efe74ffe640 1 monclient: found mon.noname-c 2024-03-29T11:23:15.249 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.270+0000 7efe74ffe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:15.249 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.270+0000 7efe74ffe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.119:6789/0 2024-03-29T11:23:15.250 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.270+0000 7efe74ffe640 1 -- 172.21.15.104:0/1140335454 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe700ac9d0 con 0x7efe700b0770 2024-03-29T11:23:15.250 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.270+0000 7efe74ffe640 10 monclient: _finish_auth 0 2024-03-29T11:23:15.250 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.270+0000 7efe74ffe640 10 monclient: _check_auth_tickets 2024-03-29T11:23:15.250 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.270+0000 7efe74ffe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:15.250 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.270+0000 7efe74ffe640 1 -- 172.21.15.104:0/1140335454 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7efe68001a30 con 0x7efe700b0770 2024-03-29T11:23:15.250 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.270+0000 7efe74ffe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:15.250 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.270+0000 7efe74ffe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:15.250 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.270+0000 7efe74ffe640 10 monclient: dump: 2024-03-29T11:23:15.250 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:23:15.250 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:23:15.250 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:15.250 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:15.250 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:15.251 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:15.251 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:23:15.251 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:23:15.251 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:23:15.251 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:15.251 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.270+0000 7efe78264640 5 monclient: authenticate success, global_id 4342 2024-03-29T11:23:15.251 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.270+0000 7efe78264640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:15.253 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.273+0000 7efe74ffe640 1 -- 172.21.15.104:0/1140335454 <== mon.1 v1:172.21.15.119:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe68001cc0 con 0x7efe700b0770 2024-03-29T11:23:15.253 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.273+0000 7efe74ffe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:15.253 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.273+0000 7efe78264640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:15.253 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.273+0000 7efe78264640 4 set_mon_vals no callback set 2024-03-29T11:23:15.253 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.273+0000 7efe74ffe640 1 -- 172.21.15.104:0/1140335454 <== mon.1 v1:172.21.15.119:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7efe68002050 con 0x7efe700b0770 2024-03-29T11:23:15.253 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.274+0000 7efe74ffe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:15.254 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.274+0000 7efe74ffe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:15.254 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.274+0000 7efe74ffe640 10 monclient: dump: 2024-03-29T11:23:15.254 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:23:15.254 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:23:15.254 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:15.254 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:15.254 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:15.254 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:15.254 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:23:15.254 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:23:15.254 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:23:15.254 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:15.254 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.274+0000 7efe78264640 10 monclient: shutdown 2024-03-29T11:23:15.254 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.274+0000 7efe78264640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:15.255 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.274+0000 7efe78264640 1 -- 172.21.15.104:0/1140335454 >> v1:172.21.15.119:6789/0 conn(0x7efe700b0770 legacy=0x7efe70149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:15.255 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.275+0000 7efe78264640 1 -- 172.21.15.104:0/1140335454 shutdown_connections 2024-03-29T11:23:15.255 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.275+0000 7efe78264640 1 -- 172.21.15.104:0/1140335454 >> 172.21.15.104:0/1140335454 conn(0x7efe7006b660 msgr2=0x7efe7006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:15.255 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.275+0000 7efe78264640 1 -- 172.21.15.104:0/1140335454 shutdown_connections 2024-03-29T11:23:15.255 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.275+0000 7efe78264640 1 -- 172.21.15.104:0/1140335454 wait complete. 2024-03-29T11:23:15.257 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.276+0000 7efe78264640 10 monclient: build_initial_monmap 2024-03-29T11:23:15.257 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.276+0000 7efe78264640 10 monclient: monmap: 2024-03-29T11:23:15.258 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:23:15.258 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:15.258 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:23:15.258 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:23:15.258 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:15.258 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:15.258 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:23:15.258 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:23:15.258 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:23:15.258 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:15.258 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.276+0000 7efe78264640 1 librados: starting msgr at 2024-03-29T11:23:15.258 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.276+0000 7efe78264640 1 librados: starting objecter 2024-03-29T11:23:15.258 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.276+0000 7efe78264640 1 Processor -- start 2024-03-29T11:23:15.258 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.277+0000 7efe78264640 1 -- start start 2024-03-29T11:23:15.259 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.277+0000 7efe78264640 1 librados: setting wanted keys 2024-03-29T11:23:15.259 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.277+0000 7efe78264640 1 librados: calling monclient init 2024-03-29T11:23:15.259 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.277+0000 7efe78264640 10 monclient: init 2024-03-29T11:23:15.259 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.277+0000 7efe78264640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:15.259 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.277+0000 7efe78264640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:23:15.259 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.277+0000 7efe78264640 10 monclient(hunting): picked mon.noname-b con 0x7efe700ac5a0 addr v1:172.21.15.119:6789/0 2024-03-29T11:23:15.259 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.277+0000 7efe78264640 10 monclient(hunting): picked mon.noname-c con 0x7efe700b0770 addr v1:172.21.15.104:6790/0 2024-03-29T11:23:15.259 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.277+0000 7efe78264640 10 monclient(hunting): picked mon.noname-a con 0x7efe7014a700 addr v1:172.21.15.104:6789/0 2024-03-29T11:23:15.259 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.277+0000 7efe78264640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe7011da80 con 0x7efe7014a700 2024-03-29T11:23:15.259 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.277+0000 7efe78264640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe701213c0 con 0x7efe700ac5a0 2024-03-29T11:23:15.259 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.277+0000 7efe78264640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe70122300 con 0x7efe700b0770 2024-03-29T11:23:15.259 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.277+0000 7efe78264640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:15.259 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.277+0000 7efe78264640 10 monclient(hunting): authenticate will time out at 1176.528869s 2024-03-29T11:23:15.260 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.282+0000 7efe757ff640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7efe700b0770 0x7efe7013fd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:35319/0 (socket says 172.21.15.104:35319) 2024-03-29T11:23:15.260 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.282+0000 7efe757ff640 1 -- 172.21.15.104:0/3856901376 learned_addr learned my addr 172.21.15.104:0/3856901376 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:23:15.264 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.285+0000 7efe6e7fc640 10 client.?.objecter ms_handle_connect 0x7efe700b0770 2024-03-29T11:23:15.264 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.285+0000 7efe6e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:15.264 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.285+0000 7efe6e7fc640 1 -- 172.21.15.104:0/3856901376 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2489651879 0 0) 0x7efe70122300 con 0x7efe700b0770 2024-03-29T11:23:15.264 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.285+0000 7efe6e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:15.264 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.285+0000 7efe6e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:15.264 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.285+0000 7efe6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:15.264 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.285+0000 7efe6e7fc640 10 monclient(hunting): my global_id is 4382 2024-03-29T11:23:15.264 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.285+0000 7efe6e7fc640 1 -- 172.21.15.104:0/3856901376 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe5c004eb0 con 0x7efe700b0770 2024-03-29T11:23:15.264 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.285+0000 7efe6e7fc640 10 client.?.objecter ms_handle_connect 0x7efe700ac5a0 2024-03-29T11:23:15.264 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.285+0000 7efe6e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:15.264 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.285+0000 7efe6e7fc640 1 -- 172.21.15.104:0/3856901376 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 905706533 0 0) 0x7efe5c004eb0 con 0x7efe700b0770 2024-03-29T11:23:15.265 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.285+0000 7efe6e7fc640 1 -- 172.21.15.104:0/3856901376 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe70122300 con 0x7efe700b0770 2024-03-29T11:23:15.265 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.285+0000 7efe6e7fc640 1 -- 172.21.15.104:0/3856901376 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7efe600017d0 con 0x7efe700b0770 2024-03-29T11:23:15.265 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.285+0000 7efe6e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:15.265 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.285+0000 7efe6e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:15.265 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.285+0000 7efe6e7fc640 10 monclient(hunting): dump: 2024-03-29T11:23:15.265 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:23:15.265 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:23:15.265 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:15.265 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:15.265 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:15.265 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:15.265 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:23:15.265 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:23:15.265 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:23:15.265 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:15.266 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.285+0000 7efe6e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:15.266 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.285+0000 7efe6e7fc640 1 -- 172.21.15.104:0/3856901376 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3390114033 0 0) 0x7efe701213c0 con 0x7efe700ac5a0 2024-03-29T11:23:15.266 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.285+0000 7efe6e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:15.266 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.285+0000 7efe6e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:15.266 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.285+0000 7efe6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:15.266 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.285+0000 7efe6e7fc640 10 monclient(hunting): my global_id is 4345 2024-03-29T11:23:15.266 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.285+0000 7efe6e7fc640 1 -- 172.21.15.104:0/3856901376 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe5c004eb0 con 0x7efe700ac5a0 2024-03-29T11:23:15.267 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.286+0000 7efe6e7fc640 10 client.?.objecter ms_handle_connect 0x7efe7014a700 2024-03-29T11:23:15.267 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.286+0000 7efe6e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:15.267 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.286+0000 7efe6e7fc640 1 -- 172.21.15.104:0/3856901376 <== mon.2 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3134777932 0 0) 0x7efe70122300 con 0x7efe700b0770 2024-03-29T11:23:15.267 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.286+0000 7efe6e7fc640 1 -- 172.21.15.104:0/3856901376 >> v1:172.21.15.119:6789/0 conn(0x7efe700ac5a0 legacy=0x7efe701217e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:15.267 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.286+0000 7efe6e7fc640 1 -- 172.21.15.104:0/3856901376 >> v1:172.21.15.104:6789/0 conn(0x7efe7014a700 legacy=0x7efe70140410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:15.267 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.286+0000 7efe6e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:23:15.267 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.286+0000 7efe6e7fc640 1 monclient: found mon.c 2024-03-29T11:23:15.268 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.286+0000 7efe6e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:15.268 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.286+0000 7efe6e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:23:15.268 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.286+0000 7efe6e7fc640 1 -- 172.21.15.104:0/3856901376 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe70123350 con 0x7efe700b0770 2024-03-29T11:23:15.268 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.286+0000 7efe6e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:23:15.268 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.286+0000 7efe6e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:23:15.268 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.286+0000 7efe6e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:15.268 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.287+0000 7efe6e7fc640 1 -- 172.21.15.104:0/3856901376 <== mon.2 v1:172.21.15.104:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe60001930 con 0x7efe700b0770 2024-03-29T11:23:15.268 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.287+0000 7efe6e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:15.268 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.287+0000 7efe6e7fc640 1 -- 172.21.15.104:0/3856901376 <== mon.2 v1:172.21.15.104:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7efe60004a00 con 0x7efe700b0770 2024-03-29T11:23:15.268 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.287+0000 7efe6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:15.268 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.287+0000 7efe6e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:15.268 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.287+0000 7efe6e7fc640 10 monclient: dump: 2024-03-29T11:23:15.268 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:23:15.268 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:23:15.268 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:15.269 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:15.269 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:15.269 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:15.269 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:23:15.269 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:23:15.269 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:23:15.269 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:15.269 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.287+0000 7efe6ffff640 4 set_mon_vals no callback set 2024-03-29T11:23:15.270 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.288+0000 7efe78264640 5 monclient: authenticate success, global_id 4382 2024-03-29T11:23:15.270 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.288+0000 7efe78264640 10 monclient: _renew_subs 2024-03-29T11:23:15.270 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.288+0000 7efe78264640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:23:15.270 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.288+0000 7efe78264640 1 -- 172.21.15.104:0/3856901376 --> v1:172.21.15.104:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe70121ef0 con 0x7efe700b0770 2024-03-29T11:23:15.270 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.288+0000 7efe78264640 10 client.4382.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:15.270 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.288+0000 7efe78264640 10 monclient: _renew_subs 2024-03-29T11:23:15.270 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.288+0000 7efe78264640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:23:15.271 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.288+0000 7efe78264640 1 -- 172.21.15.104:0/3856901376 --> v1:172.21.15.104:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efe70122a90 con 0x7efe700b0770 2024-03-29T11:23:15.271 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.288+0000 7efe78264640 1 librados: init done 2024-03-29T11:23:15.271 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.290+0000 7efe78264640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:15.271 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.290+0000 7efe78264640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:15.271 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.290+0000 7efe78264640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:23:15.271 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.290+0000 7efe78264640 1 -- 172.21.15.104:0/3856901376 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe70122490 con 0x7efe700b0770 2024-03-29T11:23:15.272 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.293+0000 7efe6e7fc640 1 -- 172.21.15.104:0/3856901376 <== mon.2 v1:172.21.15.104:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7efe6001a9d0 con 0x7efe700b0770 2024-03-29T11:23:15.272 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.293+0000 7efe6e7fc640 1 -- 172.21.15.104:0/3856901376 <== mon.2 v1:172.21.15.104:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7efe60084a30 con 0x7efe700b0770 2024-03-29T11:23:15.272 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.294+0000 7efe6e7fc640 10 client.4382.objecter ms_dispatch 0x7efe70000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:15.272 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.294+0000 7efe6e7fc640 3 client.4382.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:15.273 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.294+0000 7efe6e7fc640 3 client.4382.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:15.273 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.294+0000 7efe76000640 10 monclient: get_auth_request con 0x7efe5c06da80 auth_method 0 2024-03-29T11:23:15.273 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.294+0000 7efe6e7fc640 20 client.4382.objecter dump_active .. 0 homeless 2024-03-29T11:23:15.273 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.294+0000 7efe6e7fc640 10 client.4382.objecter ms_handle_connect 0x7efe5c06da80 2024-03-29T11:23:15.279 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.301+0000 7efe6e7fc640 1 -- 172.21.15.104:0/3856901376 <== mon.2 v1:172.21.15.104:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efe60050140 con 0x7efe700b0770 2024-03-29T11:23:15.279 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.301+0000 7efe6e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:15.279 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.301+0000 7efe6e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:15.599 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.621+0000 7efe78264640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:15.599 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.621+0000 7efe78264640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:15.600 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.621+0000 7efe78264640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:23:15.600 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.621+0000 7efe78264640 1 -- 172.21.15.104:0/3856901376 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe70122130 con 0x7efe700b0770 2024-03-29T11:23:15.604 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.625+0000 7efe6e7fc640 1 -- 172.21.15.104:0/3856901376 <== mon.2 v1:172.21.15.104:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7efe60052240 con 0x7efe700b0770 2024-03-29T11:23:15.604 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.625+0000 7efe6e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:15.604 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.625+0000 7efe6e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:15.604 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:23:15.604 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:15.607 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.629+0000 7efe78264640 10 librados: watch_flush enter 2024-03-29T11:23:15.608 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.629+0000 7efe78264640 10 librados: watch_flush exit 2024-03-29T11:23:15.608 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.629+0000 7efe78264640 20 client.4382.objecter shutdown clearing up homeless session... 2024-03-29T11:23:15.609 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.629+0000 7efe78264640 10 client.4382.objecter successfully canceled tick 2024-03-29T11:23:15.609 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.629+0000 7efe78264640 1 -- 172.21.15.104:0/3856901376 >> v1:172.21.15.104:6816/3341126660 conn(0x7efe5c06da80 legacy=0x7efe5c06ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:15.609 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.629+0000 7efe78264640 10 monclient: shutdown 2024-03-29T11:23:15.609 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.629+0000 7efe78264640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:15.609 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.629+0000 7efe78264640 1 -- 172.21.15.104:0/3856901376 >> v1:172.21.15.104:6790/0 conn(0x7efe700b0770 legacy=0x7efe7013fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:15.609 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.630+0000 7efe78264640 1 -- 172.21.15.104:0/3856901376 shutdown_connections 2024-03-29T11:23:15.609 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.630+0000 7efe78264640 1 -- 172.21.15.104:0/3856901376 >> 172.21.15.104:0/3856901376 conn(0x7efe7006b660 msgr2=0x7efe7006ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:15.609 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.630+0000 7efe78264640 1 -- 172.21.15.104:0/3856901376 shutdown_connections 2024-03-29T11:23:15.609 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.630+0000 7efe78264640 1 -- 172.21.15.104:0/3856901376 wait complete. 2024-03-29T11:23:15.609 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:15.630+0000 7efe78264640 1 librados: shutdown 2024-03-29T11:23:15.628 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:22.435 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:22.584 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.605+0000 7f5d5add4640 10 monclient: get_monmap_and_config 2024-03-29T11:23:22.584 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.605+0000 7f5d5add4640 10 monclient: build_initial_monmap 2024-03-29T11:23:22.584 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.605+0000 7f5d5add4640 10 monclient: monmap: 2024-03-29T11:23:22.584 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:23:22.584 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:23:22.584 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:23:22.606363+0000 2024-03-29T11:23:22.585 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:23:22.606363+0000 2024-03-29T11:23:22.585 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:22.585 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:22.585 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:23:22.585 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:23:22.585 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:23:22.585 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:22.585 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.606+0000 7f5d5add4640 1 Processor -- start 2024-03-29T11:23:22.585 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.606+0000 7f5d5add4640 1 -- start start 2024-03-29T11:23:22.585 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.606+0000 7f5d5add4640 10 monclient: init 2024-03-29T11:23:22.586 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.607+0000 7f5d5add4640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:22.586 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.607+0000 7f5d5add4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:23:22.586 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.607+0000 7f5d5add4640 10 monclient(hunting): picked mon.noname-a con 0x7f5d541457f0 addr v1:172.21.15.104:6789/0 2024-03-29T11:23:22.586 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.607+0000 7f5d5add4640 10 monclient(hunting): picked mon.noname-c con 0x7f5d5413f650 addr v1:172.21.15.119:6789/0 2024-03-29T11:23:22.586 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.607+0000 7f5d5add4640 10 monclient(hunting): picked mon.noname-b con 0x7f5d540a9d70 addr v1:172.21.15.104:6790/0 2024-03-29T11:23:22.586 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.607+0000 7f5d5add4640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d540aaef0 con 0x7f5d541457f0 2024-03-29T11:23:22.586 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.607+0000 7f5d5add4640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d54145bd0 con 0x7f5d5413f650 2024-03-29T11:23:22.586 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.607+0000 7f5d5add4640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d54145eb0 con 0x7f5d540a9d70 2024-03-29T11:23:22.586 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.607+0000 7f5d5add4640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:22.586 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.607+0000 7f5d5add4640 10 monclient(hunting): authenticate will time out at 1183.858491s 2024-03-29T11:23:22.591 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.613+0000 7f5d58b70640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f5d541457f0 0x7f5d5413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:55969/0 (socket says 172.21.15.104:55969) 2024-03-29T11:23:22.591 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.613+0000 7f5d58b70640 1 -- 172.21.15.104:0/2794822842 learned_addr learned my addr 172.21.15.104:0/2794822842 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:23:22.593 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.614+0000 7f5d537fe640 1 -- 172.21.15.104:0/2794822842 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2517604883 0 0) 0x7f5d540aaef0 con 0x7f5d541457f0 2024-03-29T11:23:22.593 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.614+0000 7f5d537fe640 10 monclient(hunting): _negotiate 2024-03-29T11:23:22.593 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.614+0000 7f5d537fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:22.593 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.614+0000 7f5d537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:22.594 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.614+0000 7f5d537fe640 10 monclient(hunting): my global_id is 4374 2024-03-29T11:23:22.594 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.614+0000 7f5d537fe640 1 -- 172.21.15.104:0/2794822842 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d40003520 con 0x7f5d541457f0 2024-03-29T11:23:22.594 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.616+0000 7f5d537fe640 1 -- 172.21.15.104:0/2794822842 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 857457787 0 0) 0x7f5d40003520 con 0x7f5d541457f0 2024-03-29T11:23:22.594 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.616+0000 7f5d537fe640 1 -- 172.21.15.104:0/2794822842 >> v1:172.21.15.104:6790/0 conn(0x7f5d540a9d70 legacy=0x7f5d5414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:23:22.597 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.617+0000 7f5d537fe640 1 -- 172.21.15.104:0/2794822842 >> v1:172.21.15.119:6789/0 conn(0x7f5d5413f650 legacy=0x7f5d540b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:22.597 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.617+0000 7f5d537fe640 10 monclient: _finish_hunting 0 2024-03-29T11:23:22.597 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.617+0000 7f5d537fe640 1 monclient: found mon.noname-a 2024-03-29T11:23:22.597 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.617+0000 7f5d537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:22.597 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.617+0000 7f5d537fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.104:6789/0 2024-03-29T11:23:22.598 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.617+0000 7f5d537fe640 1 -- 172.21.15.104:0/2794822842 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d54146340 con 0x7f5d541457f0 2024-03-29T11:23:22.598 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.617+0000 7f5d537fe640 10 monclient: _finish_auth 0 2024-03-29T11:23:22.598 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.617+0000 7f5d537fe640 10 monclient: _check_auth_tickets 2024-03-29T11:23:22.598 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.617+0000 7f5d537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:22.598 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.617+0000 7f5d537fe640 1 -- 172.21.15.104:0/2794822842 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f5d4c001690 con 0x7f5d541457f0 2024-03-29T11:23:22.598 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.617+0000 7f5d537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:22.598 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.617+0000 7f5d537fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:22.598 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.617+0000 7f5d537fe640 10 monclient: dump: 2024-03-29T11:23:22.598 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:23:22.598 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:23:22.598 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:22.598 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:22.598 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:22.599 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:22.599 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:23:22.599 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:23:22.599 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:23:22.599 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:22.599 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.618+0000 7f5d5add4640 5 monclient: authenticate success, global_id 4374 2024-03-29T11:23:22.599 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.618+0000 7f5d5add4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:22.599 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.618+0000 7f5d537fe640 1 -- 172.21.15.104:0/2794822842 <== mon.0 v1:172.21.15.104:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d4c001920 con 0x7f5d541457f0 2024-03-29T11:23:22.599 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.618+0000 7f5d537fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:22.599 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.618+0000 7f5d537fe640 1 -- 172.21.15.104:0/2794822842 <== mon.0 v1:172.21.15.104:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f5d4c001cf0 con 0x7f5d541457f0 2024-03-29T11:23:22.599 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.618+0000 7f5d537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:22.599 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.618+0000 7f5d537fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:22.599 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.618+0000 7f5d537fe640 10 monclient: dump: 2024-03-29T11:23:22.599 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:23:22.599 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:23:22.600 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:22.600 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:22.600 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:22.600 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:22.600 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:23:22.600 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:23:22.600 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:23:22.600 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:22.600 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.618+0000 7f5d5add4640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:22.600 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.618+0000 7f5d5add4640 4 set_mon_vals no callback set 2024-03-29T11:23:22.601 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.620+0000 7f5d5add4640 10 monclient: shutdown 2024-03-29T11:23:22.601 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.620+0000 7f5d5add4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:22.601 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.620+0000 7f5d5add4640 1 -- 172.21.15.104:0/2794822842 >> v1:172.21.15.104:6789/0 conn(0x7f5d541457f0 legacy=0x7f5d5413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:22.601 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.620+0000 7f5d5add4640 1 -- 172.21.15.104:0/2794822842 shutdown_connections 2024-03-29T11:23:22.601 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.620+0000 7f5d5add4640 1 -- 172.21.15.104:0/2794822842 >> 172.21.15.104:0/2794822842 conn(0x7f5d54066a00 msgr2=0x7f5d54066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:22.603 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.624+0000 7f5d5add4640 1 -- 172.21.15.104:0/2794822842 shutdown_connections 2024-03-29T11:23:22.605 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.626+0000 7f5d5add4640 1 -- 172.21.15.104:0/2794822842 wait complete. 2024-03-29T11:23:22.607 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.627+0000 7f5d5add4640 10 monclient: build_initial_monmap 2024-03-29T11:23:22.607 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.627+0000 7f5d5add4640 10 monclient: monmap: 2024-03-29T11:23:22.607 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:23:22.607 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:22.607 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:23:22.607 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:23:22.607 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:22.607 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:22.607 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:23:22.607 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:23:22.607 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:23:22.607 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:22.608 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.628+0000 7f5d5add4640 1 librados: starting msgr at 2024-03-29T11:23:22.608 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.628+0000 7f5d5add4640 1 librados: starting objecter 2024-03-29T11:23:22.608 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.628+0000 7f5d5add4640 1 Processor -- start 2024-03-29T11:23:22.610 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.631+0000 7f5d5add4640 1 -- start start 2024-03-29T11:23:22.610 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.632+0000 7f5d5add4640 1 librados: setting wanted keys 2024-03-29T11:23:22.610 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.632+0000 7f5d5add4640 1 librados: calling monclient init 2024-03-29T11:23:22.610 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.632+0000 7f5d5add4640 10 monclient: init 2024-03-29T11:23:22.611 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.633+0000 7f5d5add4640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:22.611 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.633+0000 7f5d5add4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:23:22.614 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.636+0000 7f5d5add4640 10 monclient(hunting): picked mon.noname-b con 0x7f5d540a9d70 addr v1:172.21.15.119:6789/0 2024-03-29T11:23:22.618 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.636+0000 7f5d5add4640 10 monclient(hunting): picked mon.noname-a con 0x7f5d5413f650 addr v1:172.21.15.104:6789/0 2024-03-29T11:23:22.618 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.638+0000 7f5d58b70640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f5d540a9d70 0x7f5d54142510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:54655/0 (socket says 172.21.15.104:54655) 2024-03-29T11:23:22.618 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.639+0000 7f5d58b70640 1 -- 172.21.15.104:0/1017513207 learned_addr learned my addr 172.21.15.104:0/1017513207 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:23:22.622 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.640+0000 7f5d5add4640 10 monclient(hunting): picked mon.noname-c con 0x7f5d5413fda0 addr v1:172.21.15.104:6790/0 2024-03-29T11:23:22.622 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.642+0000 7f5d5add4640 1 -- 172.21.15.104:0/1017513207 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d540ac740 con 0x7f5d5413f650 2024-03-29T11:23:22.622 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.641+0000 7f5d517fa640 10 client.?.objecter ms_handle_connect 0x7f5d5413f650 2024-03-29T11:23:22.622 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.642+0000 7f5d517fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:22.622 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.642+0000 7f5d517fa640 10 client.?.objecter ms_handle_connect 0x7f5d540a9d70 2024-03-29T11:23:22.622 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.642+0000 7f5d517fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:22.622 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.642+0000 7f5d5add4640 1 -- 172.21.15.104:0/1017513207 --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d54137bc0 con 0x7f5d540a9d70 2024-03-29T11:23:22.622 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.642+0000 7f5d5add4640 1 -- 172.21.15.104:0/1017513207 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d540ac9b0 con 0x7f5d5413fda0 2024-03-29T11:23:22.622 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.642+0000 7f5d5add4640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:22.622 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.642+0000 7f5d5add4640 10 monclient(hunting): authenticate will time out at 1183.893711s 2024-03-29T11:23:22.622 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.642+0000 7f5d517fa640 1 -- 172.21.15.104:0/1017513207 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1126028872 0 0) 0x7f5d54137bc0 con 0x7f5d540a9d70 2024-03-29T11:23:22.622 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.643+0000 7f5d517fa640 10 monclient(hunting): _negotiate 2024-03-29T11:23:22.622 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.643+0000 7f5d517fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:22.623 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.643+0000 7f5d517fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:22.623 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.643+0000 7f5d517fa640 10 monclient(hunting): my global_id is 4351 2024-03-29T11:23:22.623 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.643+0000 7f5d517fa640 1 -- 172.21.15.104:0/1017513207 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d34004eb0 con 0x7f5d540a9d70 2024-03-29T11:23:22.623 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.644+0000 7f5d517fa640 1 -- 172.21.15.104:0/1017513207 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1264734995 0 0) 0x7f5d540ac740 con 0x7f5d5413f650 2024-03-29T11:23:22.623 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.644+0000 7f5d517fa640 10 monclient(hunting): _negotiate 2024-03-29T11:23:22.623 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.644+0000 7f5d517fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:22.623 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.644+0000 7f5d517fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:22.623 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.644+0000 7f5d517fa640 10 monclient(hunting): my global_id is 4377 2024-03-29T11:23:22.623 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.644+0000 7f5d517fa640 1 -- 172.21.15.104:0/1017513207 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d54137bc0 con 0x7f5d5413f650 2024-03-29T11:23:22.625 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.646+0000 7f5d517fa640 1 -- 172.21.15.104:0/1017513207 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1287376477 0 0) 0x7f5d54137bc0 con 0x7f5d5413f650 2024-03-29T11:23:22.625 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.646+0000 7f5d517fa640 1 -- 172.21.15.104:0/1017513207 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d540ac740 con 0x7f5d5413f650 2024-03-29T11:23:22.625 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.646+0000 7f5d517fa640 10 client.?.objecter ms_handle_connect 0x7f5d5413fda0 2024-03-29T11:23:22.625 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.646+0000 7f5d517fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:22.625 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.646+0000 7f5d517fa640 1 -- 172.21.15.104:0/1017513207 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f5d44001b80 con 0x7f5d5413f650 2024-03-29T11:23:22.625 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.646+0000 7f5d517fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:22.625 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.646+0000 7f5d517fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:22.626 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.646+0000 7f5d517fa640 10 monclient(hunting): dump: 2024-03-29T11:23:22.626 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:23:22.626 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:23:22.626 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:22.626 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:22.626 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:22.626 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:22.626 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:23:22.626 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:23:22.626 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:23:22.626 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:22.626 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.646+0000 7f5d517fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:22.626 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.646+0000 7f5d517fa640 1 -- 172.21.15.104:0/1017513207 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4246496201 0 0) 0x7f5d34004eb0 con 0x7f5d540a9d70 2024-03-29T11:23:22.626 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.646+0000 7f5d517fa640 1 -- 172.21.15.104:0/1017513207 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d54137bc0 con 0x7f5d540a9d70 2024-03-29T11:23:22.627 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.646+0000 7f5d517fa640 1 -- 172.21.15.104:0/1017513207 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f5d4c0014b0 con 0x7f5d540a9d70 2024-03-29T11:23:22.627 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.646+0000 7f5d517fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:22.627 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.646+0000 7f5d517fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:22.627 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.646+0000 7f5d517fa640 10 monclient(hunting): dump: 2024-03-29T11:23:22.627 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:23:22.627 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:23:22.627 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:22.627 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:22.627 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:22.627 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:22.627 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:23:22.627 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:23:22.627 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:23:22.627 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:22.627 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.646+0000 7f5d517fa640 1 -- 172.21.15.104:0/1017513207 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1461084405 0 0) 0x7f5d540ac9b0 con 0x7f5d5413fda0 2024-03-29T11:23:22.627 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.647+0000 7f5d517fa640 10 monclient(hunting): _negotiate 2024-03-29T11:23:22.628 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.647+0000 7f5d517fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:22.628 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.647+0000 7f5d517fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:22.629 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.647+0000 7f5d517fa640 10 monclient(hunting): my global_id is 4385 2024-03-29T11:23:22.629 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.648+0000 7f5d517fa640 1 -- 172.21.15.104:0/1017513207 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d34004eb0 con 0x7f5d5413fda0 2024-03-29T11:23:22.629 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.648+0000 7f5d517fa640 1 -- 172.21.15.104:0/1017513207 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 633007179 0 0) 0x7f5d54137bc0 con 0x7f5d540a9d70 2024-03-29T11:23:22.629 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.648+0000 7f5d517fa640 1 -- 172.21.15.104:0/1017513207 >> v1:172.21.15.104:6790/0 conn(0x7f5d5413fda0 legacy=0x7f5d541374b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:22.629 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.648+0000 7f5d517fa640 1 -- 172.21.15.104:0/1017513207 >> v1:172.21.15.104:6789/0 conn(0x7f5d5413f650 legacy=0x7f5d54136da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:22.629 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.648+0000 7f5d517fa640 10 monclient: _finish_hunting 0 2024-03-29T11:23:22.629 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.648+0000 7f5d517fa640 1 monclient: found mon.b 2024-03-29T11:23:22.629 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.648+0000 7f5d517fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:22.629 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.648+0000 7f5d517fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:23:22.629 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.648+0000 7f5d517fa640 1 -- 172.21.15.104:0/1017513207 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d54144a40 con 0x7f5d540a9d70 2024-03-29T11:23:22.629 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.648+0000 7f5d517fa640 10 monclient: _finish_auth 0 2024-03-29T11:23:22.629 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.648+0000 7f5d517fa640 10 monclient: _check_auth_tickets 2024-03-29T11:23:22.629 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.648+0000 7f5d517fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:22.629 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.648+0000 7f5d5add4640 5 monclient: authenticate success, global_id 4351 2024-03-29T11:23:22.630 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.648+0000 7f5d5add4640 10 monclient: _renew_subs 2024-03-29T11:23:22.630 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.648+0000 7f5d5add4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:23:22.630 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.648+0000 7f5d5add4640 1 -- 172.21.15.104:0/1017513207 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d54144bd0 con 0x7f5d540a9d70 2024-03-29T11:23:22.630 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.648+0000 7f5d5add4640 10 client.4351.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:22.630 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.648+0000 7f5d5add4640 10 monclient: _renew_subs 2024-03-29T11:23:22.630 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.648+0000 7f5d5add4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:23:22.630 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.648+0000 7f5d5add4640 1 -- 172.21.15.104:0/1017513207 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d54142e00 con 0x7f5d540a9d70 2024-03-29T11:23:22.630 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.648+0000 7f5d5add4640 1 librados: init done 2024-03-29T11:23:22.630 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.649+0000 7f5d517fa640 1 -- 172.21.15.104:0/1017513207 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d4c001610 con 0x7f5d540a9d70 2024-03-29T11:23:22.630 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.649+0000 7f5d517fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:22.630 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.649+0000 7f5d52ffd640 4 set_mon_vals no callback set 2024-03-29T11:23:22.632 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.651+0000 7f5d5add4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:22.632 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.651+0000 7f5d5add4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:22.632 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.651+0000 7f5d5add4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:23:22.632 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.651+0000 7f5d5add4640 1 -- 172.21.15.104:0/1017513207 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d38005080 con 0x7f5d540a9d70 2024-03-29T11:23:22.632 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.651+0000 7f5d517fa640 1 -- 172.21.15.104:0/1017513207 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f5d4c004dd0 con 0x7f5d540a9d70 2024-03-29T11:23:22.632 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.651+0000 7f5d517fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:22.632 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.651+0000 7f5d517fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:22.632 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.651+0000 7f5d517fa640 10 monclient: dump: 2024-03-29T11:23:22.632 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:23:22.633 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:23:22.633 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:22.633 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:22.633 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:22.633 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:22.633 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:23:22.633 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:23:22.633 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:23:22.633 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:22.633 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.651+0000 7f5d517fa640 1 -- 172.21.15.104:0/1017513207 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f5d4c01ada0 con 0x7f5d540a9d70 2024-03-29T11:23:22.633 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.652+0000 7f5d517fa640 1 -- 172.21.15.104:0/1017513207 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f5d4c084b40 con 0x7f5d540a9d70 2024-03-29T11:23:22.633 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.652+0000 7f5d517fa640 10 client.4351.objecter ms_dispatch 0x7f5d54000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:22.633 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.652+0000 7f5d53fff640 10 monclient: get_auth_request con 0x7f5d3406e0c0 auth_method 0 2024-03-29T11:23:22.633 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.652+0000 7f5d517fa640 3 client.4351.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:22.633 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.652+0000 7f5d517fa640 3 client.4351.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:22.634 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.652+0000 7f5d517fa640 20 client.4351.objecter dump_active .. 0 homeless 2024-03-29T11:23:22.634 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.655+0000 7f5d517fa640 10 client.4351.objecter ms_handle_connect 0x7f5d3406e0c0 2024-03-29T11:23:22.637 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.659+0000 7f5d517fa640 1 -- 172.21.15.104:0/1017513207 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5d4c0502d0 con 0x7f5d540a9d70 2024-03-29T11:23:22.637 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.659+0000 7f5d517fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:22.637 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.659+0000 7f5d517fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:22.962 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.983+0000 7f5d5add4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:22.962 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.983+0000 7f5d5add4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:22.962 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.983+0000 7f5d5add4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:23:22.962 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.983+0000 7f5d5add4640 1 -- 172.21.15.104:0/1017513207 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d38005330 con 0x7f5d540a9d70 2024-03-29T11:23:22.962 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.984+0000 7f5d517fa640 1 -- 172.21.15.104:0/1017513207 <== mon.1 v1:172.21.15.119:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f5d4c0523d0 con 0x7f5d540a9d70 2024-03-29T11:23:22.963 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.984+0000 7f5d517fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:22.963 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.984+0000 7f5d517fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:22.965 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:23:22.965 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:22.968 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.989+0000 7f5d5add4640 10 librados: watch_flush enter 2024-03-29T11:23:22.969 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.989+0000 7f5d5add4640 10 librados: watch_flush exit 2024-03-29T11:23:22.969 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.989+0000 7f5d5add4640 20 client.4351.objecter shutdown clearing up homeless session... 2024-03-29T11:23:22.969 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.989+0000 7f5d5add4640 10 client.4351.objecter successfully canceled tick 2024-03-29T11:23:22.969 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.990+0000 7f5d5add4640 1 -- 172.21.15.104:0/1017513207 >> v1:172.21.15.104:6816/3341126660 conn(0x7f5d3406e0c0 legacy=0x7f5d34070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:22.969 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.990+0000 7f5d5add4640 10 monclient: shutdown 2024-03-29T11:23:22.969 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.990+0000 7f5d5add4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:22.969 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.990+0000 7f5d5add4640 1 -- 172.21.15.104:0/1017513207 >> v1:172.21.15.119:6789/0 conn(0x7f5d540a9d70 legacy=0x7f5d54142510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:22.970 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.990+0000 7f5d5add4640 1 -- 172.21.15.104:0/1017513207 shutdown_connections 2024-03-29T11:23:22.970 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.990+0000 7f5d5add4640 1 -- 172.21.15.104:0/1017513207 >> 172.21.15.104:0/1017513207 conn(0x7f5d54066a00 msgr2=0x7f5d540b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:22.970 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.990+0000 7f5d5add4640 1 -- 172.21.15.104:0/1017513207 shutdown_connections 2024-03-29T11:23:22.970 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.990+0000 7f5d5add4640 1 -- 172.21.15.104:0/1017513207 wait complete. 2024-03-29T11:23:22.970 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:22.990+0000 7f5d5add4640 1 librados: shutdown 2024-03-29T11:23:22.988 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:29.795 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:29.950 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.970+0000 7f66ad3c6640 10 monclient: get_monmap_and_config 2024-03-29T11:23:29.950 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.970+0000 7f66ad3c6640 10 monclient: build_initial_monmap 2024-03-29T11:23:29.950 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.970+0000 7f66ad3c6640 10 monclient: monmap: 2024-03-29T11:23:29.950 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:23:29.950 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:23:29.951 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:23:29.971505+0000 2024-03-29T11:23:29.951 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:23:29.971505+0000 2024-03-29T11:23:29.951 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:29.951 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:29.951 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:23:29.951 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:23:29.951 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:23:29.951 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:29.952 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.971+0000 7f66ad3c6640 1 Processor -- start 2024-03-29T11:23:29.952 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.971+0000 7f66ad3c6640 1 -- start start 2024-03-29T11:23:29.952 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.971+0000 7f66ad3c6640 10 monclient: init 2024-03-29T11:23:29.952 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.972+0000 7f66ad3c6640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:29.952 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.972+0000 7f66ad3c6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:23:29.952 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.972+0000 7f66ad3c6640 10 monclient(hunting): picked mon.noname-a con 0x7f66a80b0770 addr v1:172.21.15.104:6789/0 2024-03-29T11:23:29.952 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.972+0000 7f66ad3c6640 10 monclient(hunting): picked mon.noname-b con 0x7f66a814a700 addr v1:172.21.15.104:6790/0 2024-03-29T11:23:29.952 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.972+0000 7f66ad3c6640 10 monclient(hunting): picked mon.noname-c con 0x7f66a80ac5a0 addr v1:172.21.15.119:6789/0 2024-03-29T11:23:29.952 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.972+0000 7f66ad3c6640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66a80aba70 con 0x7f66a80b0770 2024-03-29T11:23:29.952 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.972+0000 7f66ad3c6640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66a80aecb0 con 0x7f66a80ac5a0 2024-03-29T11:23:29.952 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.972+0000 7f66ad3c6640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66a80af150 con 0x7f66a814a700 2024-03-29T11:23:29.953 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.972+0000 7f66ad3c6640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:29.953 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.972+0000 7f66ad3c6640 10 monclient(hunting): authenticate will time out at 1191.223788s 2024-03-29T11:23:29.957 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.977+0000 7f66a67fc640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f66a814a700 0x7f66a80abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:47675/0 (socket says 172.21.15.104:47675) 2024-03-29T11:23:29.957 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.977+0000 7f66a67fc640 1 -- 172.21.15.104:0/2893935121 learned_addr learned my addr 172.21.15.104:0/2893935121 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:23:29.959 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.980+0000 7f66a5ffb640 1 -- 172.21.15.104:0/2893935121 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 578495052 0 0) 0x7f66a80aecb0 con 0x7f66a80ac5a0 2024-03-29T11:23:29.960 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.980+0000 7f66a5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:29.960 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.980+0000 7f66a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:29.960 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.980+0000 7f66a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:29.960 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.980+0000 7f66a5ffb640 10 monclient(hunting): my global_id is 4354 2024-03-29T11:23:29.960 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.980+0000 7f66a5ffb640 1 -- 172.21.15.104:0/2893935121 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6690003520 con 0x7f66a80ac5a0 2024-03-29T11:23:29.963 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.981+0000 7f66a5ffb640 1 -- 172.21.15.104:0/2893935121 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 731079431 0 0) 0x7f66a80af150 con 0x7f66a814a700 2024-03-29T11:23:29.963 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.981+0000 7f66a5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:29.964 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.981+0000 7f66a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:29.964 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.981+0000 7f66a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:29.964 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.981+0000 7f66a5ffb640 10 monclient(hunting): my global_id is 4388 2024-03-29T11:23:29.964 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.981+0000 7f66a5ffb640 1 -- 172.21.15.104:0/2893935121 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66a80aecb0 con 0x7f66a814a700 2024-03-29T11:23:29.964 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.981+0000 7f66a5ffb640 1 -- 172.21.15.104:0/2893935121 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1315799509 0 0) 0x7f6690003520 con 0x7f66a80ac5a0 2024-03-29T11:23:29.965 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.981+0000 7f66a5ffb640 1 -- 172.21.15.104:0/2893935121 >> v1:172.21.15.104:6790/0 conn(0x7f66a814a700 legacy=0x7f66a80abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:29.965 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.981+0000 7f66a5ffb640 1 -- 172.21.15.104:0/2893935121 >> v1:172.21.15.104:6789/0 conn(0x7f66a80b0770 legacy=0x7f66a8149ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:23:29.965 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.982+0000 7f66a5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:23:29.965 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.982+0000 7f66a5ffb640 1 monclient: found mon.noname-c 2024-03-29T11:23:29.965 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.982+0000 7f66a5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:29.965 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.982+0000 7f66a5ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.119:6789/0 2024-03-29T11:23:29.966 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.982+0000 7f66a5ffb640 1 -- 172.21.15.104:0/2893935121 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a80ac9d0 con 0x7f66a80ac5a0 2024-03-29T11:23:29.966 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.982+0000 7f66a5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:23:29.966 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.982+0000 7f66a5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:23:29.966 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.982+0000 7f66a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:29.966 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.982+0000 7f66ad3c6640 5 monclient: authenticate success, global_id 4354 2024-03-29T11:23:29.967 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.982+0000 7f66ad3c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:29.967 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.982+0000 7f66a5ffb640 1 -- 172.21.15.104:0/2893935121 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f66a0001a30 con 0x7f66a80ac5a0 2024-03-29T11:23:29.967 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.982+0000 7f66a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:29.967 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.982+0000 7f66a5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:29.967 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.982+0000 7f66a5ffb640 10 monclient: dump: 2024-03-29T11:23:29.967 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:23:29.968 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:23:29.968 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:29.968 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:29.968 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:29.968 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:29.968 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:23:29.969 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:23:29.969 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:23:29.969 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:29.969 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.982+0000 7f66ad3c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:29.969 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.983+0000 7f66a5ffb640 1 -- 172.21.15.104:0/2893935121 <== mon.1 v1:172.21.15.119:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f66a0001c50 con 0x7f66a80ac5a0 2024-03-29T11:23:29.969 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.983+0000 7f66a5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:29.970 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.983+0000 7f66ad3c6640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:29.970 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.983+0000 7f66ad3c6640 4 set_mon_vals no callback set 2024-03-29T11:23:29.970 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.984+0000 7f66a5ffb640 1 -- 172.21.15.104:0/2893935121 <== mon.1 v1:172.21.15.119:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f66a0001fe0 con 0x7f66a80ac5a0 2024-03-29T11:23:29.970 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.984+0000 7f66a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:29.970 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.984+0000 7f66a5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:29.970 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.984+0000 7f66a5ffb640 10 monclient: dump: 2024-03-29T11:23:29.970 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:23:29.971 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:23:29.971 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:29.971 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:29.971 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:29.971 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:29.971 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:23:29.971 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:23:29.972 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:23:29.972 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:29.972 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.984+0000 7f66ad3c6640 10 monclient: shutdown 2024-03-29T11:23:29.972 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.984+0000 7f66ad3c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:29.972 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.984+0000 7f66ad3c6640 1 -- 172.21.15.104:0/2893935121 >> v1:172.21.15.119:6789/0 conn(0x7f66a80ac5a0 legacy=0x7f66a8151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:29.972 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.984+0000 7f66ad3c6640 1 -- 172.21.15.104:0/2893935121 shutdown_connections 2024-03-29T11:23:29.972 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.984+0000 7f66ad3c6640 1 -- 172.21.15.104:0/2893935121 >> 172.21.15.104:0/2893935121 conn(0x7f66a806b660 msgr2=0x7f66a806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:29.988 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.986+0000 7f66ad3c6640 1 -- 172.21.15.104:0/2893935121 shutdown_connections 2024-03-29T11:23:29.988 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.986+0000 7f66ad3c6640 1 -- 172.21.15.104:0/2893935121 wait complete. 2024-03-29T11:23:29.988 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.987+0000 7f66ad3c6640 10 monclient: build_initial_monmap 2024-03-29T11:23:29.988 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.987+0000 7f66ad3c6640 10 monclient: monmap: 2024-03-29T11:23:29.988 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:23:29.988 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:29.988 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:23:29.988 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:23:29.989 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:29.989 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:29.989 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:23:29.989 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:23:29.989 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:23:29.989 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:29.989 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.987+0000 7f66ad3c6640 1 librados: starting msgr at 2024-03-29T11:23:29.989 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.987+0000 7f66ad3c6640 1 librados: starting objecter 2024-03-29T11:23:29.989 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.988+0000 7f66ad3c6640 1 Processor -- start 2024-03-29T11:23:29.989 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.988+0000 7f66ad3c6640 1 -- start start 2024-03-29T11:23:29.989 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.988+0000 7f66ad3c6640 1 librados: setting wanted keys 2024-03-29T11:23:29.989 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.988+0000 7f66ad3c6640 1 librados: calling monclient init 2024-03-29T11:23:29.989 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.988+0000 7f66ad3c6640 10 monclient: init 2024-03-29T11:23:29.989 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.989+0000 7f66ad3c6640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:29.989 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.989+0000 7f66ad3c6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:23:29.989 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.989+0000 7f66ad3c6640 10 monclient(hunting): picked mon.noname-a con 0x7f66a80ac5a0 addr v1:172.21.15.104:6789/0 2024-03-29T11:23:29.990 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.989+0000 7f66ad3c6640 10 monclient(hunting): picked mon.noname-c con 0x7f66a80b0770 addr v1:172.21.15.104:6790/0 2024-03-29T11:23:29.990 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.989+0000 7f66ad3c6640 10 monclient(hunting): picked mon.noname-b con 0x7f66a814a700 addr v1:172.21.15.119:6789/0 2024-03-29T11:23:29.990 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.989+0000 7f66ad3c6640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66a81d9540 con 0x7f66a80ac5a0 2024-03-29T11:23:29.990 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.989+0000 7f66ad3c6640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66a81dbaa0 con 0x7f66a814a700 2024-03-29T11:23:29.990 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.989+0000 7f66ad3c6640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66a81271e0 con 0x7f66a80b0770 2024-03-29T11:23:29.990 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.989+0000 7f66ad3c6640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:29.990 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.989+0000 7f66ad3c6640 10 monclient(hunting): authenticate will time out at 1191.240701s 2024-03-29T11:23:29.990 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.994+0000 7f66a6ffd640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f66a80ac5a0 0x7f66a8126ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:56707/0 (socket says 172.21.15.104:56707) 2024-03-29T11:23:29.990 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.995+0000 7f66a67fc640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f66a80b0770 0x7f66a81dbfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:47759/0 (socket says 172.21.15.104:47759) 2024-03-29T11:23:29.990 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.995+0000 7f66a67fc640 1 -- 172.21.15.104:0/2583570843 learned_addr learned my addr 172.21.15.104:0/2583570843 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:23:29.990 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.995+0000 7f668f7fe640 10 client.?.objecter ms_handle_connect 0x7f66a80b0770 2024-03-29T11:23:29.990 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.995+0000 7f668f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:29.990 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.995+0000 7f668f7fe640 1 -- 172.21.15.104:0/2583570843 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2635864854 0 0) 0x7f66a81271e0 con 0x7f66a80b0770 2024-03-29T11:23:29.991 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.995+0000 7f668f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:23:29.991 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.995+0000 7f668f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:29.991 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.995+0000 7f668f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:29.991 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.995+0000 7f668f7fe640 10 monclient(hunting): my global_id is 4391 2024-03-29T11:23:29.991 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.995+0000 7f668f7fe640 1 -- 172.21.15.104:0/2583570843 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6694004eb0 con 0x7f66a80b0770 2024-03-29T11:23:29.991 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.995+0000 7f668f7fe640 10 client.?.objecter ms_handle_connect 0x7f66a80ac5a0 2024-03-29T11:23:29.991 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.995+0000 7f668f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:29.991 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.996+0000 7f668f7fe640 1 -- 172.21.15.104:0/2583570843 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1461734356 0 0) 0x7f6694004eb0 con 0x7f66a80b0770 2024-03-29T11:23:29.991 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.996+0000 7f668f7fe640 1 -- 172.21.15.104:0/2583570843 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6694005300 con 0x7f66a80b0770 2024-03-29T11:23:29.991 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.996+0000 7f668f7fe640 1 -- 172.21.15.104:0/2583570843 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f669c001910 con 0x7f66a80b0770 2024-03-29T11:23:29.991 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.996+0000 7f668f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:29.991 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.996+0000 7f668f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:29.992 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.996+0000 7f668f7fe640 10 monclient(hunting): dump: 2024-03-29T11:23:29.992 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:23:29.992 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:23:29.992 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:29.992 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:29.992 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:29.992 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:29.992 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:23:29.992 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:23:29.992 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:23:29.992 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:29.992 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.996+0000 7f668f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:29.992 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.998+0000 7f668f7fe640 1 -- 172.21.15.104:0/2583570843 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3124045758 0 0) 0x7f66a81d9540 con 0x7f66a80ac5a0 2024-03-29T11:23:29.993 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.998+0000 7f668f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:23:29.993 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.998+0000 7f668f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:29.993 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.998+0000 7f668f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:29.993 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.998+0000 7f668f7fe640 10 monclient(hunting): my global_id is 4380 2024-03-29T11:23:29.993 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.998+0000 7f668f7fe640 1 -- 172.21.15.104:0/2583570843 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6694005850 con 0x7f66a80ac5a0 2024-03-29T11:23:29.993 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.999+0000 7f668f7fe640 1 -- 172.21.15.104:0/2583570843 <== mon.2 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3255387823 0 0) 0x7f6694005300 con 0x7f66a80b0770 2024-03-29T11:23:29.993 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.999+0000 7f668f7fe640 1 -- 172.21.15.104:0/2583570843 >> v1:172.21.15.119:6789/0 conn(0x7f66a814a700 legacy=0x7f66a8144fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:29.993 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.999+0000 7f668f7fe640 1 -- 172.21.15.104:0/2583570843 >> v1:172.21.15.104:6789/0 conn(0x7f66a80ac5a0 legacy=0x7f66a8126ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:29.993 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.999+0000 7f668f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:23:29.993 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.999+0000 7f668f7fe640 1 monclient: found mon.c 2024-03-29T11:23:29.993 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.999+0000 7f668f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:29.993 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.999+0000 7f668f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:23:29.993 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.999+0000 7f668f7fe640 1 -- 172.21.15.104:0/2583570843 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a8126060 con 0x7f66a80b0770 2024-03-29T11:23:29.994 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.999+0000 7f668f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:23:29.994 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.999+0000 7f668f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:23:29.994 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.999+0000 7f668f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:29.994 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.999+0000 7f668f7fe640 1 -- 172.21.15.104:0/2583570843 <== mon.2 v1:172.21.15.104:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f669c0047e0 con 0x7f66a80b0770 2024-03-29T11:23:29.994 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.999+0000 7f668f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:29.994 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:29.999+0000 7f66a4ff9640 4 set_mon_vals no callback set 2024-03-29T11:23:29.994 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:30.000+0000 7f668f7fe640 1 -- 172.21.15.104:0/2583570843 <== mon.2 v1:172.21.15.104:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f669c001910 con 0x7f66a80b0770 2024-03-29T11:23:29.994 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:30.000+0000 7f668f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:29.994 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:30.000+0000 7f668f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:29.994 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:30.000+0000 7f668f7fe640 10 monclient: dump: 2024-03-29T11:23:29.994 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:23:29.994 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:23:29.994 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:29.995 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:29.995 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:29.995 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:29.995 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:23:29.995 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:23:29.995 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:23:29.995 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:29.995 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:30.001+0000 7f66ad3c6640 5 monclient: authenticate success, global_id 4391 2024-03-29T11:23:29.995 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:30.001+0000 7f66ad3c6640 10 monclient: _renew_subs 2024-03-29T11:23:29.995 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:30.001+0000 7f66ad3c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:23:29.995 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:30.001+0000 7f66ad3c6640 1 -- 172.21.15.104:0/2583570843 --> v1:172.21.15.104:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66a81dc6d0 con 0x7f66a80b0770 2024-03-29T11:23:29.995 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:30.001+0000 7f66ad3c6640 10 client.4391.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:29.995 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:30.002+0000 7f66ad3c6640 10 monclient: _renew_subs 2024-03-29T11:23:29.995 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:30.002+0000 7f66ad3c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:23:29.995 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:30.002+0000 7f66ad3c6640 1 -- 172.21.15.104:0/2583570843 --> v1:172.21.15.104:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f66a8127630 con 0x7f66a80b0770 2024-03-29T11:23:29.996 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:30.002+0000 7f66ad3c6640 1 librados: init done 2024-03-29T11:23:29.996 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:30.004+0000 7f668f7fe640 1 -- 172.21.15.104:0/2583570843 <== mon.2 v1:172.21.15.104:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f669c004a50 con 0x7f66a80b0770 2024-03-29T11:23:29.996 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:30.006+0000 7f668f7fe640 1 -- 172.21.15.104:0/2583570843 <== mon.2 v1:172.21.15.104:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f669c054420 con 0x7f66a80b0770 2024-03-29T11:23:29.996 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:30.006+0000 7f668f7fe640 10 client.4391.objecter ms_dispatch 0x7f66a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:29.996 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:30.007+0000 7f668f7fe640 3 client.4391.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:29.996 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:30.007+0000 7f668f7fe640 3 client.4391.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:29.996 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:30.007+0000 7f66a6ffd640 10 monclient: get_auth_request con 0x7f669406e510 auth_method 0 2024-03-29T11:23:29.996 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:30.007+0000 7f668f7fe640 20 client.4391.objecter dump_active .. 0 homeless 2024-03-29T11:23:29.996 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:30.007+0000 7f66ad3c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:29.996 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:30.007+0000 7f66ad3c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:29.996 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:30.007+0000 7f66ad3c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:23:29.996 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:30.007+0000 7f66ad3c6640 1 -- 172.21.15.104:0/2583570843 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66a80ab560 con 0x7f66a80b0770 2024-03-29T11:23:29.996 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:30.008+0000 7f668f7fe640 10 client.4391.objecter ms_handle_connect 0x7f669406e510 2024-03-29T11:23:29.997 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:30.018+0000 7f668f7fe640 1 -- 172.21.15.104:0/2583570843 <== mon.2 v1:172.21.15.104:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f669c04fd60 con 0x7f66a80b0770 2024-03-29T11:23:29.997 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:30.018+0000 7f668f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:29.997 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:30.018+0000 7f668f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:30.327 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:30.347+0000 7f66ad3c6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:30.327 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:30.347+0000 7f66ad3c6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:30.327 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:30.347+0000 7f66ad3c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:23:30.327 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:30.347+0000 7f66ad3c6640 1 -- 172.21.15.104:0/2583570843 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f66a80ab560 con 0x7f66a80b0770 2024-03-29T11:23:30.328 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:30.349+0000 7f668f7fe640 1 -- 172.21.15.104:0/2583570843 <== mon.2 v1:172.21.15.104:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f669c001770 con 0x7f66a80b0770 2024-03-29T11:23:30.329 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:30.349+0000 7f668f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:30.329 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:30.349+0000 7f668f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:30.329 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:23:30.330 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:30.333 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:30.354+0000 7f66ad3c6640 10 librados: watch_flush enter 2024-03-29T11:23:30.335 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:30.355+0000 7f66ad3c6640 10 librados: watch_flush exit 2024-03-29T11:23:30.335 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:30.355+0000 7f66ad3c6640 20 client.4391.objecter shutdown clearing up homeless session... 2024-03-29T11:23:30.335 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:30.355+0000 7f66ad3c6640 10 client.4391.objecter successfully canceled tick 2024-03-29T11:23:30.335 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:30.355+0000 7f66ad3c6640 1 -- 172.21.15.104:0/2583570843 >> v1:172.21.15.104:6816/3341126660 conn(0x7f669406e510 legacy=0x7f66940709d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:30.335 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:30.355+0000 7f66ad3c6640 10 monclient: shutdown 2024-03-29T11:23:30.336 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:30.355+0000 7f66ad3c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:30.336 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:30.355+0000 7f66ad3c6640 1 -- 172.21.15.104:0/2583570843 >> v1:172.21.15.104:6790/0 conn(0x7f66a80b0770 legacy=0x7f66a81dbfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:30.336 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:30.356+0000 7f66ad3c6640 1 -- 172.21.15.104:0/2583570843 shutdown_connections 2024-03-29T11:23:30.336 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:30.356+0000 7f66ad3c6640 1 -- 172.21.15.104:0/2583570843 >> 172.21.15.104:0/2583570843 conn(0x7f66a806b660 msgr2=0x7f66a80b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:30.336 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:30.356+0000 7f66ad3c6640 1 -- 172.21.15.104:0/2583570843 shutdown_connections 2024-03-29T11:23:30.336 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:30.356+0000 7f66ad3c6640 1 -- 172.21.15.104:0/2583570843 wait complete. 2024-03-29T11:23:30.336 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:30.356+0000 7f66ad3c6640 1 librados: shutdown 2024-03-29T11:23:30.352 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:37.157 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:37.302 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.323+0000 7ff1e1f15640 10 monclient: get_monmap_and_config 2024-03-29T11:23:37.302 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.323+0000 7ff1e1f15640 10 monclient: build_initial_monmap 2024-03-29T11:23:37.303 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.323+0000 7ff1e1f15640 10 monclient: monmap: 2024-03-29T11:23:37.303 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:23:37.303 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:23:37.303 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:23:37.324173+0000 2024-03-29T11:23:37.303 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:23:37.324173+0000 2024-03-29T11:23:37.303 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:37.303 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:37.303 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:23:37.303 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:23:37.303 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:23:37.303 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:37.304 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.324+0000 7ff1e1f15640 1 Processor -- start 2024-03-29T11:23:37.304 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.325+0000 7ff1e1f15640 1 -- start start 2024-03-29T11:23:37.304 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.325+0000 7ff1e1f15640 10 monclient: init 2024-03-29T11:23:37.305 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.325+0000 7ff1e1f15640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:37.305 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.325+0000 7ff1e1f15640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:23:37.305 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.326+0000 7ff1e1f15640 10 monclient(hunting): picked mon.noname-b con 0x7ff1dc149a90 addr v1:172.21.15.104:6790/0 2024-03-29T11:23:37.306 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.326+0000 7ff1e1f15640 10 monclient(hunting): picked mon.noname-c con 0x7ff1dc149e70 addr v1:172.21.15.119:6789/0 2024-03-29T11:23:37.306 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.326+0000 7ff1e1f15640 10 monclient(hunting): picked mon.noname-a con 0x7ff1dc0af740 addr v1:172.21.15.104:6789/0 2024-03-29T11:23:37.306 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.326+0000 7ff1e1f15640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1dc14a3d0 con 0x7ff1dc0af740 2024-03-29T11:23:37.306 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.326+0000 7ff1e1f15640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1dc072730 con 0x7ff1dc149e70 2024-03-29T11:23:37.306 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.326+0000 7ff1e1f15640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1dc071f90 con 0x7ff1dc149a90 2024-03-29T11:23:37.306 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.326+0000 7ff1e1f15640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:37.306 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.326+0000 7ff1e1f15640 10 monclient(hunting): authenticate will time out at 1198.577086s 2024-03-29T11:23:37.306 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.326+0000 7ff1dbfff640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7ff1dc149e70 0x7ff1dc06be90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:60644/0 (socket says 172.21.15.104:60644) 2024-03-29T11:23:37.306 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.326+0000 7ff1dbfff640 1 -- 172.21.15.104:0/3185262999 learned_addr learned my addr 172.21.15.104:0/3185262999 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:23:37.309 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.327+0000 7ff1db7fe640 1 -- 172.21.15.104:0/3185262999 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3487075135 0 0) 0x7ff1dc071f90 con 0x7ff1dc149a90 2024-03-29T11:23:37.309 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.327+0000 7ff1db7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:23:37.309 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.327+0000 7ff1db7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:37.309 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.327+0000 7ff1db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:37.309 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.327+0000 7ff1db7fe640 10 monclient(hunting): my global_id is 4394 2024-03-29T11:23:37.309 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.327+0000 7ff1db7fe640 1 -- 172.21.15.104:0/3185262999 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1cc003520 con 0x7ff1dc149a90 2024-03-29T11:23:37.309 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.327+0000 7ff1db7fe640 1 -- 172.21.15.104:0/3185262999 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1103799421 0 0) 0x7ff1dc072730 con 0x7ff1dc149e70 2024-03-29T11:23:37.309 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.327+0000 7ff1db7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:23:37.309 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.327+0000 7ff1db7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:37.309 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.327+0000 7ff1db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:37.309 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.327+0000 7ff1db7fe640 10 monclient(hunting): my global_id is 4357 2024-03-29T11:23:37.309 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.327+0000 7ff1db7fe640 1 -- 172.21.15.104:0/3185262999 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1dc071f90 con 0x7ff1dc149e70 2024-03-29T11:23:37.310 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.327+0000 7ff1db7fe640 1 -- 172.21.15.104:0/3185262999 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3238007792 0 0) 0x7ff1cc003520 con 0x7ff1dc149a90 2024-03-29T11:23:37.310 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.327+0000 7ff1db7fe640 1 -- 172.21.15.104:0/3185262999 >> v1:172.21.15.119:6789/0 conn(0x7ff1dc149e70 legacy=0x7ff1dc06be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:37.310 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.327+0000 7ff1db7fe640 1 -- 172.21.15.104:0/3185262999 >> v1:172.21.15.104:6789/0 conn(0x7ff1dc0af740 legacy=0x7ff1dc06c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:37.310 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.327+0000 7ff1db7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:23:37.310 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.327+0000 7ff1db7fe640 1 monclient: found mon.noname-b 2024-03-29T11:23:37.310 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.327+0000 7ff1db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:37.310 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.327+0000 7ff1db7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.104:6790/0 2024-03-29T11:23:37.310 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.327+0000 7ff1db7fe640 1 -- 172.21.15.104:0/3185262999 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1dc072120 con 0x7ff1dc149a90 2024-03-29T11:23:37.310 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.327+0000 7ff1db7fe640 10 monclient: _finish_auth 0 2024-03-29T11:23:37.310 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.327+0000 7ff1db7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:23:37.310 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.327+0000 7ff1db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:37.310 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.327+0000 7ff1e1f15640 5 monclient: authenticate success, global_id 4394 2024-03-29T11:23:37.310 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.327+0000 7ff1e1f15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:37.310 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.327+0000 7ff1db7fe640 1 -- 172.21.15.104:0/3185262999 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7ff1d4001a10 con 0x7ff1dc149a90 2024-03-29T11:23:37.310 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.327+0000 7ff1db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:37.311 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.327+0000 7ff1db7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:37.311 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.327+0000 7ff1db7fe640 10 monclient: dump: 2024-03-29T11:23:37.311 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:23:37.311 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:23:37.311 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:37.311 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:37.311 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:37.311 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:37.311 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:23:37.311 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:23:37.311 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:23:37.311 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:37.311 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.327+0000 7ff1e1f15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:37.311 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.328+0000 7ff1db7fe640 1 -- 172.21.15.104:0/3185262999 <== mon.2 v1:172.21.15.104:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1d4001c30 con 0x7ff1dc149a90 2024-03-29T11:23:37.312 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.328+0000 7ff1db7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:37.312 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.328+0000 7ff1db7fe640 1 -- 172.21.15.104:0/3185262999 <== mon.2 v1:172.21.15.104:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7ff1d4001f40 con 0x7ff1dc149a90 2024-03-29T11:23:37.312 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.328+0000 7ff1db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:37.312 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.328+0000 7ff1db7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:37.312 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.328+0000 7ff1db7fe640 10 monclient: dump: 2024-03-29T11:23:37.312 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:23:37.312 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:23:37.312 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:37.312 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:37.312 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:37.312 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:37.312 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:23:37.312 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:23:37.312 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:23:37.312 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:37.313 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.328+0000 7ff1e1f15640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:37.313 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.328+0000 7ff1e1f15640 4 set_mon_vals no callback set 2024-03-29T11:23:37.313 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.329+0000 7ff1e1f15640 10 monclient: shutdown 2024-03-29T11:23:37.313 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.329+0000 7ff1e1f15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:37.313 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.329+0000 7ff1e1f15640 1 -- 172.21.15.104:0/3185262999 >> v1:172.21.15.104:6790/0 conn(0x7ff1dc149a90 legacy=0x7ff1dc071880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:37.313 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.329+0000 7ff1e1f15640 1 -- 172.21.15.104:0/3185262999 shutdown_connections 2024-03-29T11:23:37.313 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.329+0000 7ff1e1f15640 1 -- 172.21.15.104:0/3185262999 >> 172.21.15.104:0/3185262999 conn(0x7ff1dc0671c0 msgr2=0x7ff1dc0b59b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:37.313 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.329+0000 7ff1e1f15640 1 -- 172.21.15.104:0/3185262999 shutdown_connections 2024-03-29T11:23:37.313 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.329+0000 7ff1e1f15640 1 -- 172.21.15.104:0/3185262999 wait complete. 2024-03-29T11:23:37.319 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.330+0000 7ff1e1f15640 10 monclient: build_initial_monmap 2024-03-29T11:23:37.319 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.331+0000 7ff1e1f15640 10 monclient: monmap: 2024-03-29T11:23:37.319 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:23:37.319 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:37.319 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:23:37.319 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:23:37.319 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:37.319 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:37.319 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:23:37.319 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:23:37.319 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:23:37.319 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:37.319 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.331+0000 7ff1e1f15640 1 librados: starting msgr at 2024-03-29T11:23:37.319 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.331+0000 7ff1e1f15640 1 librados: starting objecter 2024-03-29T11:23:37.319 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.331+0000 7ff1e1f15640 1 Processor -- start 2024-03-29T11:23:37.320 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.332+0000 7ff1e1f15640 1 -- start start 2024-03-29T11:23:37.320 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.332+0000 7ff1e1f15640 1 librados: setting wanted keys 2024-03-29T11:23:37.320 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.332+0000 7ff1e1f15640 1 librados: calling monclient init 2024-03-29T11:23:37.320 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.332+0000 7ff1e1f15640 10 monclient: init 2024-03-29T11:23:37.320 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.333+0000 7ff1e1f15640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:37.320 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.333+0000 7ff1e1f15640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:23:37.320 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.333+0000 7ff1e1f15640 10 monclient(hunting): picked mon.noname-a con 0x7ff1dc0af740 addr v1:172.21.15.104:6789/0 2024-03-29T11:23:37.320 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.333+0000 7ff1e1f15640 10 monclient(hunting): picked mon.noname-c con 0x7ff1dc149a90 addr v1:172.21.15.104:6790/0 2024-03-29T11:23:37.320 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.333+0000 7ff1e0f13640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7ff1dc0af740 0x7ff1dc1dbdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:50564/0 (socket says 172.21.15.104:50564) 2024-03-29T11:23:37.320 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.333+0000 7ff1e0f13640 1 -- 172.21.15.104:0/3463658160 learned_addr learned my addr 172.21.15.104:0/3463658160 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:23:37.320 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.333+0000 7ff1e1f15640 10 monclient(hunting): picked mon.noname-b con 0x7ff1dc149e70 addr v1:172.21.15.119:6789/0 2024-03-29T11:23:37.320 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.333+0000 7ff1e1f15640 1 -- 172.21.15.104:0/3463658160 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1dc1db930 con 0x7ff1dc0af740 2024-03-29T11:23:37.320 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.333+0000 7ff1d97fa640 10 client.?.objecter ms_handle_connect 0x7ff1dc0af740 2024-03-29T11:23:37.320 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.333+0000 7ff1d97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:37.320 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.333+0000 7ff1e1f15640 1 -- 172.21.15.104:0/3463658160 --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1dc1d8fd0 con 0x7ff1dc149e70 2024-03-29T11:23:37.321 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.333+0000 7ff1e1f15640 1 -- 172.21.15.104:0/3463658160 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1dc1374c0 con 0x7ff1dc149a90 2024-03-29T11:23:37.321 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.333+0000 7ff1e1f15640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:37.321 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.333+0000 7ff1d97fa640 10 client.?.objecter ms_handle_connect 0x7ff1dc149e70 2024-03-29T11:23:37.321 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.333+0000 7ff1d97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:37.321 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.333+0000 7ff1e1f15640 10 monclient(hunting): authenticate will time out at 1198.584805s 2024-03-29T11:23:37.321 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.334+0000 7ff1d97fa640 10 client.?.objecter ms_handle_connect 0x7ff1dc149a90 2024-03-29T11:23:37.321 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.334+0000 7ff1d97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:37.321 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.334+0000 7ff1d97fa640 1 -- 172.21.15.104:0/3463658160 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4064023741 0 0) 0x7ff1dc1d8fd0 con 0x7ff1dc149e70 2024-03-29T11:23:37.321 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.334+0000 7ff1d97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:23:37.321 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.334+0000 7ff1d97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:37.321 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.334+0000 7ff1d97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:37.321 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.334+0000 7ff1d97fa640 10 monclient(hunting): my global_id is 4360 2024-03-29T11:23:37.321 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.334+0000 7ff1d97fa640 1 -- 172.21.15.104:0/3463658160 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1c0004eb0 con 0x7ff1dc149e70 2024-03-29T11:23:37.321 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.334+0000 7ff1d97fa640 1 -- 172.21.15.104:0/3463658160 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1443941809 0 0) 0x7ff1dc1db930 con 0x7ff1dc0af740 2024-03-29T11:23:37.322 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.334+0000 7ff1d97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:23:37.322 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.334+0000 7ff1d97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:37.322 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.334+0000 7ff1d97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:37.322 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.334+0000 7ff1d97fa640 10 monclient(hunting): my global_id is 4383 2024-03-29T11:23:37.322 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.334+0000 7ff1d97fa640 1 -- 172.21.15.104:0/3463658160 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1dc1d8fd0 con 0x7ff1dc0af740 2024-03-29T11:23:37.322 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.334+0000 7ff1d97fa640 1 -- 172.21.15.104:0/3463658160 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2974765361 0 0) 0x7ff1c0004eb0 con 0x7ff1dc149e70 2024-03-29T11:23:37.322 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.334+0000 7ff1d97fa640 1 -- 172.21.15.104:0/3463658160 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1dc1db930 con 0x7ff1dc149e70 2024-03-29T11:23:37.322 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.334+0000 7ff1d97fa640 1 -- 172.21.15.104:0/3463658160 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7ff1d00017e0 con 0x7ff1dc149e70 2024-03-29T11:23:37.322 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.334+0000 7ff1d97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:37.322 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.334+0000 7ff1d97fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:37.322 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.334+0000 7ff1d97fa640 10 monclient(hunting): dump: 2024-03-29T11:23:37.322 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:23:37.322 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:23:37.322 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:37.323 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:37.323 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:37.323 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:37.323 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:23:37.323 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:23:37.323 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:23:37.323 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:37.323 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.334+0000 7ff1d97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:37.323 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.334+0000 7ff1d97fa640 1 -- 172.21.15.104:0/3463658160 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1897349888 0 0) 0x7ff1dc1d8fd0 con 0x7ff1dc0af740 2024-03-29T11:23:37.323 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.335+0000 7ff1d97fa640 1 -- 172.21.15.104:0/3463658160 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1c0004eb0 con 0x7ff1dc0af740 2024-03-29T11:23:37.323 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.335+0000 7ff1d97fa640 1 -- 172.21.15.104:0/3463658160 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7ff1d4001990 con 0x7ff1dc0af740 2024-03-29T11:23:37.323 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.335+0000 7ff1d97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:37.323 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.335+0000 7ff1d97fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:37.323 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.335+0000 7ff1d97fa640 10 monclient(hunting): dump: 2024-03-29T11:23:37.323 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:23:37.323 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:23:37.324 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:37.324 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:37.324 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:37.324 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:37.324 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:23:37.324 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:23:37.324 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:23:37.324 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:37.324 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.335+0000 7ff1d97fa640 1 -- 172.21.15.104:0/3463658160 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2464404284 0 0) 0x7ff1dc1374c0 con 0x7ff1dc149a90 2024-03-29T11:23:37.324 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.335+0000 7ff1d97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:23:37.324 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.335+0000 7ff1d97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:37.324 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.335+0000 7ff1d97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:37.324 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.335+0000 7ff1d97fa640 10 monclient(hunting): my global_id is 4397 2024-03-29T11:23:37.324 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.335+0000 7ff1d97fa640 1 -- 172.21.15.104:0/3463658160 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1dc1d8fd0 con 0x7ff1dc149a90 2024-03-29T11:23:37.324 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.335+0000 7ff1d97fa640 1 -- 172.21.15.104:0/3463658160 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1526904040 0 0) 0x7ff1dc1db930 con 0x7ff1dc149e70 2024-03-29T11:23:37.325 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.335+0000 7ff1d97fa640 1 -- 172.21.15.104:0/3463658160 >> v1:172.21.15.104:6790/0 conn(0x7ff1dc149a90 legacy=0x7ff1dc1d7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:37.325 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.335+0000 7ff1d97fa640 1 -- 172.21.15.104:0/3463658160 >> v1:172.21.15.104:6789/0 conn(0x7ff1dc0af740 legacy=0x7ff1dc1dbdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:37.325 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.335+0000 7ff1d97fa640 10 monclient: _finish_hunting 0 2024-03-29T11:23:37.325 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.335+0000 7ff1d97fa640 1 monclient: found mon.b 2024-03-29T11:23:37.325 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.335+0000 7ff1d97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:37.325 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.335+0000 7ff1d97fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:23:37.325 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.335+0000 7ff1d97fa640 1 -- 172.21.15.104:0/3463658160 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1dc1db410 con 0x7ff1dc149e70 2024-03-29T11:23:37.325 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.335+0000 7ff1d97fa640 10 monclient: _finish_auth 0 2024-03-29T11:23:37.325 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.335+0000 7ff1d97fa640 10 monclient: _check_auth_tickets 2024-03-29T11:23:37.325 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.335+0000 7ff1d97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:37.325 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.335+0000 7ff1e1f15640 5 monclient: authenticate success, global_id 4360 2024-03-29T11:23:37.325 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.335+0000 7ff1e1f15640 10 monclient: _renew_subs 2024-03-29T11:23:37.325 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.335+0000 7ff1e1f15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:23:37.325 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.335+0000 7ff1e1f15640 1 -- 172.21.15.104:0/3463658160 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1dc13f240 con 0x7ff1dc149e70 2024-03-29T11:23:37.325 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.335+0000 7ff1e1f15640 10 client.4360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:37.326 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.335+0000 7ff1e1f15640 10 monclient: _renew_subs 2024-03-29T11:23:37.326 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.335+0000 7ff1e1f15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:23:37.326 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.335+0000 7ff1e1f15640 1 -- 172.21.15.104:0/3463658160 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1dc137f80 con 0x7ff1dc149e70 2024-03-29T11:23:37.326 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.335+0000 7ff1e1f15640 1 librados: init done 2024-03-29T11:23:37.326 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.336+0000 7ff1d97fa640 1 -- 172.21.15.104:0/3463658160 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1d0001d10 con 0x7ff1dc149e70 2024-03-29T11:23:37.326 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.336+0000 7ff1d97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:37.326 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.336+0000 7ff1d97fa640 1 -- 172.21.15.104:0/3463658160 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7ff1d0003790 con 0x7ff1dc149e70 2024-03-29T11:23:37.326 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.336+0000 7ff1d97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:37.326 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.336+0000 7ff1d97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:37.326 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.336+0000 7ff1d97fa640 10 monclient: dump: 2024-03-29T11:23:37.326 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:23:37.326 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:23:37.326 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:37.326 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:37.327 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:37.327 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:37.327 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:23:37.327 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:23:37.327 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:23:37.327 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:37.327 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.336+0000 7ff1daffd640 4 set_mon_vals no callback set 2024-03-29T11:23:37.327 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.337+0000 7ff1e1f15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:37.327 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.338+0000 7ff1d97fa640 1 -- 172.21.15.104:0/3463658160 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7ff1d00038f0 con 0x7ff1dc149e70 2024-03-29T11:23:37.327 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.338+0000 7ff1e1f15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:37.327 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.338+0000 7ff1e1f15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:23:37.327 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.338+0000 7ff1e1f15640 1 -- 172.21.15.104:0/3463658160 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1bc005080 con 0x7ff1dc149e70 2024-03-29T11:23:37.327 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.338+0000 7ff1d97fa640 1 -- 172.21.15.104:0/3463658160 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7ff1d0083790 con 0x7ff1dc149e70 2024-03-29T11:23:37.327 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.338+0000 7ff1d97fa640 10 client.4360.objecter ms_dispatch 0x7ff1dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:37.327 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.338+0000 7ff1d97fa640 3 client.4360.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:37.328 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.338+0000 7ff1d97fa640 3 client.4360.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:37.328 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.338+0000 7ff1d97fa640 20 client.4360.objecter dump_active .. 0 homeless 2024-03-29T11:23:37.328 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.339+0000 7ff1e0f13640 10 monclient: get_auth_request con 0x7ff1c006df90 auth_method 0 2024-03-29T11:23:37.328 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.339+0000 7ff1d97fa640 10 client.4360.objecter ms_handle_connect 0x7ff1c006df90 2024-03-29T11:23:37.328 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.346+0000 7ff1d97fa640 1 -- 172.21.15.104:0/3463658160 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff1d004f090 con 0x7ff1dc149e70 2024-03-29T11:23:37.329 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.346+0000 7ff1d97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:37.329 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.346+0000 7ff1d97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:37.728 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.749+0000 7ff1e1f15640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:37.728 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.749+0000 7ff1e1f15640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:37.729 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.749+0000 7ff1e1f15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:23:37.729 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.749+0000 7ff1e1f15640 1 -- 172.21.15.104:0/3463658160 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff1bc005330 con 0x7ff1dc149e70 2024-03-29T11:23:37.730 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.750+0000 7ff1d97fa640 1 -- 172.21.15.104:0/3463658160 <== mon.1 v1:172.21.15.119:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7ff1d0050fc0 con 0x7ff1dc149e70 2024-03-29T11:23:37.730 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.750+0000 7ff1d97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:37.730 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.750+0000 7ff1d97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:37.730 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:23:37.730 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:37.735 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.754+0000 7ff1e1f15640 10 librados: watch_flush enter 2024-03-29T11:23:37.735 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.754+0000 7ff1e1f15640 10 librados: watch_flush exit 2024-03-29T11:23:37.735 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.754+0000 7ff1e1f15640 20 client.4360.objecter shutdown clearing up homeless session... 2024-03-29T11:23:37.735 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.754+0000 7ff1e1f15640 10 client.4360.objecter successfully canceled tick 2024-03-29T11:23:37.735 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.754+0000 7ff1e1f15640 1 -- 172.21.15.104:0/3463658160 >> v1:172.21.15.104:6816/3341126660 conn(0x7ff1c006df90 legacy=0x7ff1c0070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:37.735 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.754+0000 7ff1e1f15640 10 monclient: shutdown 2024-03-29T11:23:37.735 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.754+0000 7ff1e1f15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:37.735 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.754+0000 7ff1e1f15640 1 -- 172.21.15.104:0/3463658160 >> v1:172.21.15.119:6789/0 conn(0x7ff1dc149e70 legacy=0x7ff1dc1d88c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:37.735 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.755+0000 7ff1e1f15640 1 -- 172.21.15.104:0/3463658160 shutdown_connections 2024-03-29T11:23:37.735 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.755+0000 7ff1e1f15640 1 -- 172.21.15.104:0/3463658160 >> 172.21.15.104:0/3463658160 conn(0x7ff1dc0671c0 msgr2=0x7ff1dc0b2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:37.735 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.755+0000 7ff1e1f15640 1 -- 172.21.15.104:0/3463658160 shutdown_connections 2024-03-29T11:23:37.735 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.755+0000 7ff1e1f15640 1 -- 172.21.15.104:0/3463658160 wait complete. 2024-03-29T11:23:37.736 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:37.755+0000 7ff1e1f15640 1 librados: shutdown 2024-03-29T11:23:37.750 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:44.557 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:44.712 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.732+0000 7f470e9a3640 10 monclient: get_monmap_and_config 2024-03-29T11:23:44.712 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.732+0000 7f470e9a3640 10 monclient: build_initial_monmap 2024-03-29T11:23:44.712 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.732+0000 7f470e9a3640 10 monclient: monmap: 2024-03-29T11:23:44.712 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:23:44.712 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:23:44.713 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:23:44.733084+0000 2024-03-29T11:23:44.713 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:23:44.733084+0000 2024-03-29T11:23:44.713 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:44.713 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:44.713 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:23:44.713 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:23:44.713 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:23:44.713 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:44.714 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.733+0000 7f470e9a3640 1 Processor -- start 2024-03-29T11:23:44.714 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.733+0000 7f470e9a3640 1 -- start start 2024-03-29T11:23:44.714 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.733+0000 7f470e9a3640 10 monclient: init 2024-03-29T11:23:44.714 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.734+0000 7f470e9a3640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:44.714 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.734+0000 7f470e9a3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:23:44.714 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.734+0000 7f470e9a3640 10 monclient(hunting): picked mon.noname-a con 0x7f47080a9750 addr v1:172.21.15.104:6789/0 2024-03-29T11:23:44.714 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.734+0000 7f470e9a3640 10 monclient(hunting): picked mon.noname-c con 0x7f47080adca0 addr v1:172.21.15.119:6789/0 2024-03-29T11:23:44.714 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.734+0000 7f470e9a3640 10 monclient(hunting): picked mon.noname-b con 0x7f4708146a80 addr v1:172.21.15.104:6790/0 2024-03-29T11:23:44.714 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.734+0000 7f470e9a3640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47080abdc0 con 0x7f47080a9750 2024-03-29T11:23:44.715 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.734+0000 7f470e9a3640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47080b0070 con 0x7f47080adca0 2024-03-29T11:23:44.715 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.734+0000 7f470e9a3640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47080af880 con 0x7f4708146a80 2024-03-29T11:23:44.715 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.734+0000 7f470e9a3640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:44.715 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.734+0000 7f470e9a3640 10 monclient(hunting): authenticate will time out at 1205.985914s 2024-03-29T11:23:44.719 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.739+0000 7f470e1a2640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f4708146a80 0x7f4708151bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:39435/0 (socket says 172.21.15.104:39435) 2024-03-29T11:23:44.719 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.740+0000 7f470e1a2640 1 -- 172.21.15.104:0/3196138055 learned_addr learned my addr 172.21.15.104:0/3196138055 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:23:44.725 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.743+0000 7f470c99f640 1 -- 172.21.15.104:0/3196138055 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 520064654 0 0) 0x7f47080af880 con 0x7f4708146a80 2024-03-29T11:23:44.725 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.743+0000 7f470c99f640 10 monclient(hunting): _negotiate 2024-03-29T11:23:44.725 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.743+0000 7f470c99f640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:44.725 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.743+0000 7f470c99f640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:44.725 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.743+0000 7f470c99f640 10 monclient(hunting): my global_id is 4400 2024-03-29T11:23:44.725 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.743+0000 7f470c99f640 1 -- 172.21.15.104:0/3196138055 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46f4003520 con 0x7f4708146a80 2024-03-29T11:23:44.726 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.743+0000 7f470c99f640 1 -- 172.21.15.104:0/3196138055 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2016991926 0 0) 0x7f47080abdc0 con 0x7f47080a9750 2024-03-29T11:23:44.726 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.743+0000 7f470c99f640 10 monclient(hunting): _negotiate 2024-03-29T11:23:44.726 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.743+0000 7f470c99f640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:44.726 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.743+0000 7f470c99f640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:44.726 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.743+0000 7f470c99f640 10 monclient(hunting): my global_id is 4386 2024-03-29T11:23:44.726 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.743+0000 7f470c99f640 1 -- 172.21.15.104:0/3196138055 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47080af880 con 0x7f47080a9750 2024-03-29T11:23:44.726 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.743+0000 7f470c99f640 1 -- 172.21.15.104:0/3196138055 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1602812867 0 0) 0x7f47080b0070 con 0x7f47080adca0 2024-03-29T11:23:44.726 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.743+0000 7f470c99f640 10 monclient(hunting): _negotiate 2024-03-29T11:23:44.726 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.743+0000 7f470c99f640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:44.726 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.743+0000 7f470c99f640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:44.726 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.743+0000 7f470c99f640 10 monclient(hunting): my global_id is 4363 2024-03-29T11:23:44.726 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.743+0000 7f470c99f640 1 -- 172.21.15.104:0/3196138055 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47080abdc0 con 0x7f47080adca0 2024-03-29T11:23:44.726 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.743+0000 7f470c99f640 1 -- 172.21.15.104:0/3196138055 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 229693196 0 0) 0x7f47080abdc0 con 0x7f47080adca0 2024-03-29T11:23:44.727 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.743+0000 7f470c99f640 1 -- 172.21.15.104:0/3196138055 >> v1:172.21.15.104:6790/0 conn(0x7f4708146a80 legacy=0x7f4708151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:44.727 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.743+0000 7f470c99f640 1 -- 172.21.15.104:0/3196138055 >> v1:172.21.15.104:6789/0 conn(0x7f47080a9750 legacy=0x7f4708143710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:44.727 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.743+0000 7f470c99f640 10 monclient: _finish_hunting 0 2024-03-29T11:23:44.727 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.743+0000 7f470c99f640 1 monclient: found mon.noname-c 2024-03-29T11:23:44.727 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.743+0000 7f470c99f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:44.727 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.743+0000 7f470c99f640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.119:6789/0 2024-03-29T11:23:44.727 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.743+0000 7f470c99f640 1 -- 172.21.15.104:0/3196138055 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47080ac030 con 0x7f47080adca0 2024-03-29T11:23:44.727 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.743+0000 7f470c99f640 10 monclient: _finish_auth 0 2024-03-29T11:23:44.727 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.743+0000 7f470c99f640 10 monclient: _check_auth_tickets 2024-03-29T11:23:44.727 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.743+0000 7f470c99f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:44.727 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.743+0000 7f470c99f640 1 -- 172.21.15.104:0/3196138055 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f4700001930 con 0x7f47080adca0 2024-03-29T11:23:44.727 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.743+0000 7f470c99f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:44.727 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.743+0000 7f470c99f640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:44.727 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.743+0000 7f470c99f640 10 monclient: dump: 2024-03-29T11:23:44.728 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:23:44.728 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:23:44.728 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:44.728 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:44.728 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:44.728 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:44.728 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:23:44.728 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:23:44.728 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:23:44.728 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:44.728 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.744+0000 7f470c99f640 1 -- 172.21.15.104:0/3196138055 <== mon.1 v1:172.21.15.119:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4700001b10 con 0x7f47080adca0 2024-03-29T11:23:44.728 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.744+0000 7f470c99f640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:44.728 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.744+0000 7f470c99f640 1 -- 172.21.15.104:0/3196138055 <== mon.1 v1:172.21.15.119:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f4700001930 con 0x7f47080adca0 2024-03-29T11:23:44.728 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.744+0000 7f470c99f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:44.728 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.744+0000 7f470c99f640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:44.728 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.744+0000 7f470c99f640 10 monclient: dump: 2024-03-29T11:23:44.729 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:23:44.729 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:23:44.729 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:44.729 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:44.729 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:44.729 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:44.729 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:23:44.729 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:23:44.729 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:23:44.729 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:44.729 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.744+0000 7f470e9a3640 5 monclient: authenticate success, global_id 4363 2024-03-29T11:23:44.729 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.744+0000 7f470e9a3640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:44.729 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.744+0000 7f470e9a3640 4 set_mon_vals no callback set 2024-03-29T11:23:44.729 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.745+0000 7f470e9a3640 10 monclient: shutdown 2024-03-29T11:23:44.729 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.745+0000 7f470e9a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:44.729 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.745+0000 7f470e9a3640 1 -- 172.21.15.104:0/3196138055 >> v1:172.21.15.119:6789/0 conn(0x7f47080adca0 legacy=0x7f4708146370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:44.730 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.748+0000 7f470e9a3640 1 -- 172.21.15.104:0/3196138055 shutdown_connections 2024-03-29T11:23:44.730 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.748+0000 7f470e9a3640 1 -- 172.21.15.104:0/3196138055 >> 172.21.15.104:0/3196138055 conn(0x7f47080673e0 msgr2=0x7f47080b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:44.730 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.750+0000 7f470e9a3640 1 -- 172.21.15.104:0/3196138055 shutdown_connections 2024-03-29T11:23:44.731 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.751+0000 7f470e9a3640 1 -- 172.21.15.104:0/3196138055 wait complete. 2024-03-29T11:23:44.733 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.752+0000 7f470e9a3640 10 monclient: build_initial_monmap 2024-03-29T11:23:44.733 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.752+0000 7f470e9a3640 10 monclient: monmap: 2024-03-29T11:23:44.733 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:23:44.733 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:44.733 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:23:44.733 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:23:44.734 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:44.734 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:44.734 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:23:44.734 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:23:44.734 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:23:44.734 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:44.734 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.753+0000 7f470e9a3640 1 librados: starting msgr at 2024-03-29T11:23:44.734 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.753+0000 7f470e9a3640 1 librados: starting objecter 2024-03-29T11:23:44.734 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.753+0000 7f470e9a3640 1 Processor -- start 2024-03-29T11:23:44.734 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.753+0000 7f470e9a3640 1 -- start start 2024-03-29T11:23:44.734 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.753+0000 7f470e9a3640 1 librados: setting wanted keys 2024-03-29T11:23:44.734 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.753+0000 7f470e9a3640 1 librados: calling monclient init 2024-03-29T11:23:44.734 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.753+0000 7f470e9a3640 10 monclient: init 2024-03-29T11:23:44.735 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.754+0000 7f470e9a3640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:44.736 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.754+0000 7f470e9a3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:23:44.736 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.754+0000 7f470e9a3640 10 monclient(hunting): picked mon.noname-b con 0x7f47080a9750 addr v1:172.21.15.119:6789/0 2024-03-29T11:23:44.736 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.754+0000 7f470e9a3640 10 monclient(hunting): picked mon.noname-a con 0x7f47080adca0 addr v1:172.21.15.104:6789/0 2024-03-29T11:23:44.736 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.754+0000 7f470e9a3640 10 monclient(hunting): picked mon.noname-c con 0x7f4708146a80 addr v1:172.21.15.104:6790/0 2024-03-29T11:23:44.736 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.754+0000 7f470e9a3640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47080722b0 con 0x7f47080adca0 2024-03-29T11:23:44.736 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.754+0000 7f470e9a3640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47080725b0 con 0x7f47080a9750 2024-03-29T11:23:44.736 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.754+0000 7f470e9a3640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4708071e60 con 0x7f4708146a80 2024-03-29T11:23:44.736 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.754+0000 7f470e9a3640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:44.736 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.754+0000 7f470e9a3640 10 monclient(hunting): authenticate will time out at 1206.005742s 2024-03-29T11:23:44.737 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.757+0000 7f470d9a1640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f47080a9750 0x7f47081df5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:34723/0 (socket says 172.21.15.104:34723) 2024-03-29T11:23:44.737 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.757+0000 7f470d9a1640 1 -- 172.21.15.104:0/4206373308 learned_addr learned my addr 172.21.15.104:0/4206373308 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:23:44.737 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.757+0000 7f46fe7fc640 10 client.?.objecter ms_handle_connect 0x7f47080a9750 2024-03-29T11:23:44.737 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.757+0000 7f46fe7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:44.739 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.758+0000 7f46fe7fc640 1 -- 172.21.15.104:0/4206373308 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1012088008 0 0) 0x7f47080725b0 con 0x7f47080a9750 2024-03-29T11:23:44.739 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.758+0000 7f46fe7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:44.739 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.758+0000 7f46fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:44.739 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.758+0000 7f46fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:44.739 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.758+0000 7f46fe7fc640 10 monclient(hunting): my global_id is 4366 2024-03-29T11:23:44.739 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.758+0000 7f46fe7fc640 1 -- 172.21.15.104:0/4206373308 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46f4004ae0 con 0x7f47080a9750 2024-03-29T11:23:44.739 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.759+0000 7f46fe7fc640 1 -- 172.21.15.104:0/4206373308 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1245111313 0 0) 0x7f46f4004ae0 con 0x7f47080a9750 2024-03-29T11:23:44.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.759+0000 7f46fe7fc640 1 -- 172.21.15.104:0/4206373308 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47080725b0 con 0x7f47080a9750 2024-03-29T11:23:44.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.759+0000 7f46fe7fc640 1 -- 172.21.15.104:0/4206373308 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f46f80019b0 con 0x7f47080a9750 2024-03-29T11:23:44.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.759+0000 7f46fe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:44.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.759+0000 7f46fe7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:44.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.759+0000 7f46fe7fc640 10 monclient(hunting): dump: 2024-03-29T11:23:44.740 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:23:44.740 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:23:44.740 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:44.740 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:44.740 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:44.740 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:44.740 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:23:44.740 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:23:44.740 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:23:44.740 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:44.741 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.759+0000 7f46fe7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:44.741 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.759+0000 7f46fe7fc640 1 -- 172.21.15.104:0/4206373308 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 237108939 0 0) 0x7f47080725b0 con 0x7f47080a9750 2024-03-29T11:23:44.741 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.759+0000 7f46fe7fc640 1 -- 172.21.15.104:0/4206373308 >> v1:172.21.15.104:6790/0 conn(0x7f4708146a80 legacy=0x7f47081e6150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:23:44.741 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.760+0000 7f46fe7fc640 1 -- 172.21.15.104:0/4206373308 >> v1:172.21.15.104:6789/0 conn(0x7f47080adca0 legacy=0x7f470806e470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:23:44.742 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.762+0000 7f46fe7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:23:44.742 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.762+0000 7f46fe7fc640 1 monclient: found mon.b 2024-03-29T11:23:44.742 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.762+0000 7f46fe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:44.742 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.762+0000 7f46fe7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:23:44.742 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.762+0000 7f46fe7fc640 1 -- 172.21.15.104:0/4206373308 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4708071ff0 con 0x7f47080a9750 2024-03-29T11:23:44.742 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.762+0000 7f46fe7fc640 10 monclient: _finish_auth 0 2024-03-29T11:23:44.742 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.762+0000 7f46fe7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:23:44.742 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.762+0000 7f46fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:44.742 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.762+0000 7f470e9a3640 5 monclient: authenticate success, global_id 4366 2024-03-29T11:23:44.742 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.762+0000 7f470e9a3640 10 monclient: _renew_subs 2024-03-29T11:23:44.742 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.762+0000 7f470e9a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:23:44.742 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.762+0000 7f470e9a3640 1 -- 172.21.15.104:0/4206373308 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f470813dfd0 con 0x7f47080a9750 2024-03-29T11:23:44.743 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.762+0000 7f470e9a3640 10 client.4366.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:44.743 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.762+0000 7f470e9a3640 10 monclient: _renew_subs 2024-03-29T11:23:44.743 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.762+0000 7f470e9a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:23:44.743 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.762+0000 7f470e9a3640 1 -- 172.21.15.104:0/4206373308 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f47081dee80 con 0x7f47080a9750 2024-03-29T11:23:44.743 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.762+0000 7f470e9a3640 1 librados: init done 2024-03-29T11:23:44.744 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.763+0000 7f470e9a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:44.744 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.763+0000 7f470e9a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:44.744 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.763+0000 7f470e9a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:23:44.744 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.763+0000 7f470e9a3640 1 -- 172.21.15.104:0/4206373308 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f470804f730 con 0x7f47080a9750 2024-03-29T11:23:44.744 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.763+0000 7f46fe7fc640 1 -- 172.21.15.104:0/4206373308 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46f8001cb0 con 0x7f47080a9750 2024-03-29T11:23:44.744 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.763+0000 7f46fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:44.744 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.763+0000 7f46fe7fc640 1 -- 172.21.15.104:0/4206373308 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f46f8003a60 con 0x7f47080a9750 2024-03-29T11:23:44.744 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.763+0000 7f46fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:44.744 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.763+0000 7f46fe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:44.744 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.763+0000 7f46fe7fc640 10 monclient: dump: 2024-03-29T11:23:44.744 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:23:44.745 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:23:44.745 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:44.745 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:44.745 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:44.745 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:44.745 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:23:44.745 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:23:44.745 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:23:44.745 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:44.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.765+0000 7f46fe7fc640 1 -- 172.21.15.104:0/4206373308 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f46f8003bc0 con 0x7f47080a9750 2024-03-29T11:23:44.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.765+0000 7f46fe7fc640 1 -- 172.21.15.104:0/4206373308 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f46f8084a50 con 0x7f47080a9750 2024-03-29T11:23:44.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.765+0000 7f46fe7fc640 10 client.4366.objecter ms_dispatch 0x7f4708000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:44.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.765+0000 7f46fe7fc640 3 client.4366.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:44.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.765+0000 7f46fe7fc640 3 client.4366.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:44.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.766+0000 7f46fe7fc640 20 client.4366.objecter dump_active .. 0 homeless 2024-03-29T11:23:44.747 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.766+0000 7f470d1a0640 10 monclient: get_auth_request con 0x7f46f406e000 auth_method 0 2024-03-29T11:23:44.747 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.766+0000 7f46fe7fc640 10 client.4366.objecter ms_handle_connect 0x7f46f406e000 2024-03-29T11:23:44.747 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.768+0000 7f46fffff640 4 set_mon_vals no callback set 2024-03-29T11:23:44.752 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.773+0000 7f46fe7fc640 1 -- 172.21.15.104:0/4206373308 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f46f8050330 con 0x7f47080a9750 2024-03-29T11:23:44.752 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.773+0000 7f46fe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:44.752 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:44.773+0000 7f46fe7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:45.106 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:45.127+0000 7f470e9a3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:45.107 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:45.127+0000 7f470e9a3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:45.107 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:45.127+0000 7f470e9a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:23:45.107 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:45.127+0000 7f470e9a3640 1 -- 172.21.15.104:0/4206373308 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4708074e10 con 0x7f47080a9750 2024-03-29T11:23:45.107 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:45.128+0000 7f46fe7fc640 1 -- 172.21.15.104:0/4206373308 <== mon.1 v1:172.21.15.119:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f46f8052260 con 0x7f47080a9750 2024-03-29T11:23:45.108 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:45.128+0000 7f46fe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:45.108 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:45.128+0000 7f46fe7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:45.108 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:23:45.108 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:45.111 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:45.131+0000 7f470e9a3640 10 librados: watch_flush enter 2024-03-29T11:23:45.111 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:45.132+0000 7f470e9a3640 10 librados: watch_flush exit 2024-03-29T11:23:45.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:45.132+0000 7f470e9a3640 20 client.4366.objecter shutdown clearing up homeless session... 2024-03-29T11:23:45.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:45.132+0000 7f470e9a3640 10 client.4366.objecter successfully canceled tick 2024-03-29T11:23:45.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:45.132+0000 7f470e9a3640 1 -- 172.21.15.104:0/4206373308 >> v1:172.21.15.104:6816/3341126660 conn(0x7f46f406e000 legacy=0x7f46f40704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:45.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:45.132+0000 7f470e9a3640 10 monclient: shutdown 2024-03-29T11:23:45.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:45.132+0000 7f470e9a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:45.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:45.132+0000 7f470e9a3640 1 -- 172.21.15.104:0/4206373308 >> v1:172.21.15.119:6789/0 conn(0x7f47080a9750 legacy=0x7f47081df5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:45.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:45.132+0000 7f470e9a3640 1 -- 172.21.15.104:0/4206373308 shutdown_connections 2024-03-29T11:23:45.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:45.132+0000 7f470e9a3640 1 -- 172.21.15.104:0/4206373308 >> 172.21.15.104:0/4206373308 conn(0x7f47080673e0 msgr2=0x7f47080aa200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:45.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:45.132+0000 7f470e9a3640 1 -- 172.21.15.104:0/4206373308 shutdown_connections 2024-03-29T11:23:45.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:45.133+0000 7f470e9a3640 1 -- 172.21.15.104:0/4206373308 wait complete. 2024-03-29T11:23:45.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:45.133+0000 7f470e9a3640 1 librados: shutdown 2024-03-29T11:23:45.128 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:51.935 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:52.083 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.102+0000 7f146f79d640 10 monclient: get_monmap_and_config 2024-03-29T11:23:52.083 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.102+0000 7f146f79d640 10 monclient: build_initial_monmap 2024-03-29T11:23:52.083 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.102+0000 7f146f79d640 10 monclient: monmap: 2024-03-29T11:23:52.083 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:23:52.083 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:23:52.083 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:23:52.103748+0000 2024-03-29T11:23:52.083 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:23:52.103748+0000 2024-03-29T11:23:52.083 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:52.083 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:52.083 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:23:52.083 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:23:52.084 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:23:52.084 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:52.086 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.104+0000 7f146f79d640 1 Processor -- start 2024-03-29T11:23:52.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.104+0000 7f146f79d640 1 -- start start 2024-03-29T11:23:52.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.104+0000 7f146f79d640 10 monclient: init 2024-03-29T11:23:52.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.105+0000 7f146f79d640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:52.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.105+0000 7f146f79d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:23:52.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.105+0000 7f146f79d640 10 monclient(hunting): picked mon.noname-a con 0x7f14680b1f60 addr v1:172.21.15.104:6789/0 2024-03-29T11:23:52.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.105+0000 7f146f79d640 10 monclient(hunting): picked mon.noname-c con 0x7f1468071eb0 addr v1:172.21.15.119:6789/0 2024-03-29T11:23:52.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.105+0000 7f146f79d640 10 monclient(hunting): picked mon.noname-b con 0x7f1468143d10 addr v1:172.21.15.104:6790/0 2024-03-29T11:23:52.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.105+0000 7f146f79d640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f146806ef80 con 0x7f14680b1f60 2024-03-29T11:23:52.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.105+0000 7f146f79d640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f146806f260 con 0x7f1468071eb0 2024-03-29T11:23:52.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.105+0000 7f146f79d640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14680b40c0 con 0x7f1468143d10 2024-03-29T11:23:52.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.105+0000 7f146f79d640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:52.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.105+0000 7f146f79d640 10 monclient(hunting): authenticate will time out at 1213.356671s 2024-03-29T11:23:52.088 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.107+0000 7f146dd3a640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f1468143d10 0x7f1468146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:44182/0 (socket says 172.21.15.104:44182) 2024-03-29T11:23:52.088 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.107+0000 7f146dd3a640 1 -- 172.21.15.104:0/1822721005 learned_addr learned my addr 172.21.15.104:0/1822721005 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:23:52.094 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.110+0000 7f145ffff640 1 -- 172.21.15.104:0/1822721005 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2711206109 0 0) 0x7f14680b40c0 con 0x7f1468143d10 2024-03-29T11:23:52.094 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.110+0000 7f145ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:52.094 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.110+0000 7f145ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:52.095 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.110+0000 7f145ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:52.095 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.110+0000 7f145ffff640 10 monclient(hunting): my global_id is 4403 2024-03-29T11:23:52.095 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.110+0000 7f145ffff640 1 -- 172.21.15.104:0/1822721005 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1454003520 con 0x7f1468143d10 2024-03-29T11:23:52.095 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.110+0000 7f145ffff640 1 -- 172.21.15.104:0/1822721005 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3098662520 0 0) 0x7f146806f260 con 0x7f1468071eb0 2024-03-29T11:23:52.095 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.110+0000 7f145ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:52.095 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.110+0000 7f145ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:52.095 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.110+0000 7f145ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:52.095 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.110+0000 7f145ffff640 10 monclient(hunting): my global_id is 4369 2024-03-29T11:23:52.095 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.110+0000 7f145ffff640 1 -- 172.21.15.104:0/1822721005 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14680b40c0 con 0x7f1468071eb0 2024-03-29T11:23:52.095 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.111+0000 7f145ffff640 1 -- 172.21.15.104:0/1822721005 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3877889131 0 0) 0x7f146806ef80 con 0x7f14680b1f60 2024-03-29T11:23:52.095 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.111+0000 7f145ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:52.095 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.111+0000 7f145ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:52.096 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.111+0000 7f145ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:52.096 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.111+0000 7f145ffff640 10 monclient(hunting): my global_id is 4389 2024-03-29T11:23:52.096 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.111+0000 7f145ffff640 1 -- 172.21.15.104:0/1822721005 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f146806f260 con 0x7f14680b1f60 2024-03-29T11:23:52.096 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.111+0000 7f145ffff640 1 -- 172.21.15.104:0/1822721005 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2086414385 0 0) 0x7f1454003520 con 0x7f1468143d10 2024-03-29T11:23:52.096 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.111+0000 7f145ffff640 1 -- 172.21.15.104:0/1822721005 >> v1:172.21.15.119:6789/0 conn(0x7f1468071eb0 legacy=0x7f1468146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:52.096 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.111+0000 7f145ffff640 1 -- 172.21.15.104:0/1822721005 >> v1:172.21.15.104:6789/0 conn(0x7f14680b1f60 legacy=0x7f1468143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:52.096 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.111+0000 7f145ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:23:52.096 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.111+0000 7f145ffff640 1 monclient: found mon.noname-b 2024-03-29T11:23:52.096 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.111+0000 7f145ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:52.096 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.111+0000 7f145ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.104:6790/0 2024-03-29T11:23:52.097 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.111+0000 7f145ffff640 1 -- 172.21.15.104:0/1822721005 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1468155d70 con 0x7f1468143d10 2024-03-29T11:23:52.097 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.111+0000 7f145ffff640 10 monclient: _finish_auth 0 2024-03-29T11:23:52.097 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.111+0000 7f145ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:23:52.097 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.111+0000 7f145ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:52.097 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.111+0000 7f145ffff640 1 -- 172.21.15.104:0/1822721005 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f1458001690 con 0x7f1468143d10 2024-03-29T11:23:52.097 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.111+0000 7f145ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:52.097 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.111+0000 7f145ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:52.097 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.111+0000 7f145ffff640 10 monclient: dump: 2024-03-29T11:23:52.097 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:23:52.097 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:23:52.097 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:52.097 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:52.097 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:52.098 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:52.098 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:23:52.098 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:23:52.098 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:23:52.098 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:52.098 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.111+0000 7f146f79d640 5 monclient: authenticate success, global_id 4403 2024-03-29T11:23:52.098 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.111+0000 7f146f79d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:52.098 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.111+0000 7f145ffff640 1 -- 172.21.15.104:0/1822721005 <== mon.2 v1:172.21.15.104:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f14580018b0 con 0x7f1468143d10 2024-03-29T11:23:52.098 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.111+0000 7f145ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:52.099 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.111+0000 7f145ffff640 1 -- 172.21.15.104:0/1822721005 <== mon.2 v1:172.21.15.104:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f1458001c40 con 0x7f1468143d10 2024-03-29T11:23:52.099 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.111+0000 7f146f79d640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:52.099 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.111+0000 7f146f79d640 4 set_mon_vals no callback set 2024-03-29T11:23:52.099 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.111+0000 7f145ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:52.099 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.111+0000 7f145ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:52.099 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.111+0000 7f145ffff640 10 monclient: dump: 2024-03-29T11:23:52.099 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:23:52.099 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:23:52.099 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:52.100 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:52.100 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:52.100 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:52.100 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:23:52.100 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:23:52.100 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:23:52.100 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:52.100 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.112+0000 7f146f79d640 10 monclient: shutdown 2024-03-29T11:23:52.100 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.112+0000 7f146f79d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:52.100 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.112+0000 7f146f79d640 1 -- 172.21.15.104:0/1822721005 >> v1:172.21.15.104:6790/0 conn(0x7f1468143d10 legacy=0x7f1468146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:52.101 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.112+0000 7f146f79d640 1 -- 172.21.15.104:0/1822721005 shutdown_connections 2024-03-29T11:23:52.101 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.112+0000 7f146f79d640 1 -- 172.21.15.104:0/1822721005 >> 172.21.15.104:0/1822721005 conn(0x7f1468066a00 msgr2=0x7f1468066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:52.101 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.113+0000 7f146f79d640 1 -- 172.21.15.104:0/1822721005 shutdown_connections 2024-03-29T11:23:52.110 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.113+0000 7f146f79d640 1 -- 172.21.15.104:0/1822721005 wait complete. 2024-03-29T11:23:52.110 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.114+0000 7f146f79d640 10 monclient: build_initial_monmap 2024-03-29T11:23:52.110 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.114+0000 7f146f79d640 10 monclient: monmap: 2024-03-29T11:23:52.110 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:23:52.110 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:52.110 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:23:52.111 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:23:52.111 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:52.111 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:52.111 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:23:52.111 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:23:52.111 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:23:52.111 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:52.111 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.116+0000 7f146f79d640 1 librados: starting msgr at 2024-03-29T11:23:52.111 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.116+0000 7f146f79d640 1 librados: starting objecter 2024-03-29T11:23:52.111 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.116+0000 7f146f79d640 1 Processor -- start 2024-03-29T11:23:52.111 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.116+0000 7f146f79d640 1 -- start start 2024-03-29T11:23:52.111 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.116+0000 7f146f79d640 1 librados: setting wanted keys 2024-03-29T11:23:52.111 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.116+0000 7f146f79d640 1 librados: calling monclient init 2024-03-29T11:23:52.111 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.116+0000 7f146f79d640 10 monclient: init 2024-03-29T11:23:52.111 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.117+0000 7f146f79d640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:52.111 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.117+0000 7f146f79d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:23:52.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.117+0000 7f146f79d640 10 monclient(hunting): picked mon.noname-a con 0x7f14680b1f60 addr v1:172.21.15.104:6789/0 2024-03-29T11:23:52.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.119+0000 7f146f79d640 10 monclient(hunting): picked mon.noname-b con 0x7f1468143d10 addr v1:172.21.15.119:6789/0 2024-03-29T11:23:52.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.119+0000 7f146f79d640 10 monclient(hunting): picked mon.noname-c con 0x7f1468070b40 addr v1:172.21.15.104:6790/0 2024-03-29T11:23:52.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.119+0000 7f146f79d640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f146814b280 con 0x7f14680b1f60 2024-03-29T11:23:52.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.119+0000 7f146f79d640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f146814b4d0 con 0x7f1468143d10 2024-03-29T11:23:52.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.119+0000 7f146f79d640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f146814b7b0 con 0x7f1468070b40 2024-03-29T11:23:52.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.119+0000 7f146f79d640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:52.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.119+0000 7f146f79d640 10 monclient(hunting): authenticate will time out at 1213.370972s 2024-03-29T11:23:52.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.120+0000 7f146dd3a640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f1468070b40 0x7f14681e6370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:46812/0 (socket says 172.21.15.104:46812) 2024-03-29T11:23:52.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.120+0000 7f146dd3a640 1 -- 172.21.15.104:0/2158593875 learned_addr learned my addr 172.21.15.104:0/2158593875 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:23:52.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.120+0000 7f145dffb640 10 client.?.objecter ms_handle_connect 0x7f1468070b40 2024-03-29T11:23:52.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.120+0000 7f145dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:52.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.120+0000 7f145dffb640 1 -- 172.21.15.104:0/2158593875 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3012392550 0 0) 0x7f146814b7b0 con 0x7f1468070b40 2024-03-29T11:23:52.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.120+0000 7f145dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:52.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.120+0000 7f145dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:52.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.120+0000 7f145dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:52.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.120+0000 7f145dffb640 10 monclient(hunting): my global_id is 4406 2024-03-29T11:23:52.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.120+0000 7f145dffb640 1 -- 172.21.15.104:0/2158593875 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f144c004eb0 con 0x7f1468070b40 2024-03-29T11:23:52.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.123+0000 7f145dffb640 10 client.?.objecter ms_handle_connect 0x7f1468143d10 2024-03-29T11:23:52.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.123+0000 7f145dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:52.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.123+0000 7f145dffb640 10 client.?.objecter ms_handle_connect 0x7f14680b1f60 2024-03-29T11:23:52.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.123+0000 7f145dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:52.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.123+0000 7f145dffb640 1 -- 172.21.15.104:0/2158593875 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3659935018 0 0) 0x7f146814b4d0 con 0x7f1468143d10 2024-03-29T11:23:52.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.123+0000 7f145dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:52.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.123+0000 7f145dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:52.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.123+0000 7f145dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:52.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.123+0000 7f145dffb640 10 monclient(hunting): my global_id is 4372 2024-03-29T11:23:52.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.123+0000 7f145dffb640 1 -- 172.21.15.104:0/2158593875 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f146814b7b0 con 0x7f1468143d10 2024-03-29T11:23:52.114 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.123+0000 7f145dffb640 1 -- 172.21.15.104:0/2158593875 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1097582601 0 0) 0x7f144c004eb0 con 0x7f1468070b40 2024-03-29T11:23:52.114 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.123+0000 7f145dffb640 1 -- 172.21.15.104:0/2158593875 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f146814b4d0 con 0x7f1468070b40 2024-03-29T11:23:52.114 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.123+0000 7f145dffb640 1 -- 172.21.15.104:0/2158593875 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f1458001bc0 con 0x7f1468070b40 2024-03-29T11:23:52.114 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.123+0000 7f145dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:52.114 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.123+0000 7f145dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:52.114 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.123+0000 7f145dffb640 10 monclient(hunting): dump: 2024-03-29T11:23:52.114 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:23:52.114 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:23:52.114 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:52.114 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:52.114 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:52.114 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:52.114 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:23:52.114 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:23:52.115 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:23:52.115 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:52.115 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.123+0000 7f145dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:52.115 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.123+0000 7f145dffb640 1 -- 172.21.15.104:0/2158593875 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3666358620 0 0) 0x7f146814b280 con 0x7f14680b1f60 2024-03-29T11:23:52.115 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.123+0000 7f145dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:52.115 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.123+0000 7f145dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:52.115 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.123+0000 7f145dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:52.115 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.123+0000 7f145dffb640 10 monclient(hunting): my global_id is 4392 2024-03-29T11:23:52.115 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.123+0000 7f145dffb640 1 -- 172.21.15.104:0/2158593875 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f144c004eb0 con 0x7f14680b1f60 2024-03-29T11:23:52.115 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.124+0000 7f145dffb640 1 -- 172.21.15.104:0/2158593875 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3574546360 0 0) 0x7f146814b7b0 con 0x7f1468143d10 2024-03-29T11:23:52.115 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.124+0000 7f145dffb640 1 -- 172.21.15.104:0/2158593875 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f144c005ab0 con 0x7f1468143d10 2024-03-29T11:23:52.115 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.124+0000 7f145dffb640 1 -- 172.21.15.104:0/2158593875 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f1460001d00 con 0x7f1468143d10 2024-03-29T11:23:52.115 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.124+0000 7f145dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:52.115 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.124+0000 7f145dffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:52.115 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.124+0000 7f145dffb640 10 monclient(hunting): dump: 2024-03-29T11:23:52.115 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:23:52.116 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:23:52.116 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:52.116 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:52.116 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:52.116 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:52.116 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:23:52.116 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:23:52.116 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:23:52.116 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:52.116 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.124+0000 7f145dffb640 1 -- 172.21.15.104:0/2158593875 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 178386332 0 0) 0x7f144c004eb0 con 0x7f14680b1f60 2024-03-29T11:23:52.116 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.124+0000 7f145dffb640 1 -- 172.21.15.104:0/2158593875 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f146814b7b0 con 0x7f14680b1f60 2024-03-29T11:23:52.116 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.124+0000 7f145dffb640 1 -- 172.21.15.104:0/2158593875 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f1464001930 con 0x7f14680b1f60 2024-03-29T11:23:52.116 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.124+0000 7f145dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:52.116 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.124+0000 7f145dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:52.117 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.124+0000 7f145dffb640 10 monclient(hunting): dump: 2024-03-29T11:23:52.117 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:23:52.117 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:23:52.117 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:52.117 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:52.117 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:52.117 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:52.117 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:23:52.117 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:23:52.117 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:23:52.117 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:52.117 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.124+0000 7f145dffb640 1 -- 172.21.15.104:0/2158593875 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 64828122 0 0) 0x7f144c005ab0 con 0x7f1468143d10 2024-03-29T11:23:52.117 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.124+0000 7f145dffb640 1 -- 172.21.15.104:0/2158593875 >> v1:172.21.15.104:6790/0 conn(0x7f1468070b40 legacy=0x7f14681e6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:52.117 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.124+0000 7f145dffb640 1 -- 172.21.15.104:0/2158593875 >> v1:172.21.15.104:6789/0 conn(0x7f14680b1f60 legacy=0x7f14681dee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:52.117 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.124+0000 7f145dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:23:52.118 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.124+0000 7f145dffb640 1 monclient: found mon.b 2024-03-29T11:23:52.118 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.124+0000 7f145dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:52.118 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.124+0000 7f145dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:23:52.118 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.124+0000 7f145dffb640 1 -- 172.21.15.104:0/2158593875 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f146813fdc0 con 0x7f1468143d10 2024-03-29T11:23:52.118 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.124+0000 7f145dffb640 10 monclient: _finish_auth 0 2024-03-29T11:23:52.118 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.124+0000 7f145dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:23:52.118 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.124+0000 7f145dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:52.118 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.124+0000 7f146f79d640 5 monclient: authenticate success, global_id 4372 2024-03-29T11:23:52.118 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.124+0000 7f146f79d640 10 monclient: _renew_subs 2024-03-29T11:23:52.118 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.124+0000 7f146f79d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:23:52.118 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.124+0000 7f146f79d640 1 -- 172.21.15.104:0/2158593875 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1468140070 con 0x7f1468143d10 2024-03-29T11:23:52.118 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.125+0000 7f146f79d640 10 client.4372.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:52.118 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.125+0000 7f146f79d640 10 monclient: _renew_subs 2024-03-29T11:23:52.118 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.125+0000 7f146f79d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:23:52.118 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.125+0000 7f146f79d640 1 -- 172.21.15.104:0/2158593875 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f14681e7c70 con 0x7f1468143d10 2024-03-29T11:23:52.119 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.125+0000 7f146f79d640 1 librados: init done 2024-03-29T11:23:52.119 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.125+0000 7f145dffb640 1 -- 172.21.15.104:0/2158593875 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1460003970 con 0x7f1468143d10 2024-03-29T11:23:52.119 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.125+0000 7f145dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:52.119 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.125+0000 7f145f7fe640 4 set_mon_vals no callback set 2024-03-29T11:23:52.119 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.125+0000 7f145dffb640 1 -- 172.21.15.104:0/2158593875 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f1460003c80 con 0x7f1468143d10 2024-03-29T11:23:52.119 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.125+0000 7f145dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:52.119 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.125+0000 7f145dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:52.119 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.126+0000 7f145dffb640 10 monclient: dump: 2024-03-29T11:23:52.119 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:23:52.119 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:23:52.119 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:52.119 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:52.119 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:52.119 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:52.119 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:23:52.119 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:23:52.120 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:23:52.120 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:52.120 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.127+0000 7f146f79d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:52.120 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.127+0000 7f146f79d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:52.120 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.127+0000 7f146f79d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:23:52.120 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.127+0000 7f146f79d640 1 -- 172.21.15.104:0/2158593875 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1448005080 con 0x7f1468143d10 2024-03-29T11:23:52.120 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.127+0000 7f145dffb640 1 -- 172.21.15.104:0/2158593875 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f1460019c50 con 0x7f1468143d10 2024-03-29T11:23:52.120 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.128+0000 7f145dffb640 1 -- 172.21.15.104:0/2158593875 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f1460083b30 con 0x7f1468143d10 2024-03-29T11:23:52.120 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.128+0000 7f145dffb640 10 client.4372.objecter ms_dispatch 0x7f1468000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:52.120 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.128+0000 7f145dffb640 3 client.4372.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:52.120 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.128+0000 7f145dffb640 3 client.4372.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:52.120 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.128+0000 7f145dffb640 20 client.4372.objecter dump_active .. 0 homeless 2024-03-29T11:23:52.120 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.128+0000 7f146d539640 10 monclient: get_auth_request con 0x7f144c06e3b0 auth_method 0 2024-03-29T11:23:52.121 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.129+0000 7f145dffb640 10 client.4372.objecter ms_handle_connect 0x7f144c06e3b0 2024-03-29T11:23:52.121 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.134+0000 7f145dffb640 1 -- 172.21.15.104:0/2158593875 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f146004f410 con 0x7f1468143d10 2024-03-29T11:23:52.121 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.135+0000 7f145dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:52.121 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.135+0000 7f145dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:52.441 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.460+0000 7f146f79d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:52.442 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.460+0000 7f146f79d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:52.442 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.460+0000 7f146f79d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:23:52.442 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.460+0000 7f146f79d640 1 -- 172.21.15.104:0/2158593875 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1448005330 con 0x7f1468143d10 2024-03-29T11:23:52.442 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.461+0000 7f145dffb640 1 -- 172.21.15.104:0/2158593875 <== mon.1 v1:172.21.15.119:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f1460051340 con 0x7f1468143d10 2024-03-29T11:23:52.442 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.461+0000 7f145dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:52.442 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.461+0000 7f145dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:52.443 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:23:52.443 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:52.445 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.464+0000 7f146f79d640 10 librados: watch_flush enter 2024-03-29T11:23:52.445 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.464+0000 7f146f79d640 10 librados: watch_flush exit 2024-03-29T11:23:52.445 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.464+0000 7f146f79d640 20 client.4372.objecter shutdown clearing up homeless session... 2024-03-29T11:23:52.445 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.464+0000 7f146f79d640 10 client.4372.objecter successfully canceled tick 2024-03-29T11:23:52.446 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.464+0000 7f146f79d640 1 -- 172.21.15.104:0/2158593875 >> v1:172.21.15.104:6816/3341126660 conn(0x7f144c06e3b0 legacy=0x7f144c070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:52.446 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.464+0000 7f146f79d640 10 monclient: shutdown 2024-03-29T11:23:52.446 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.464+0000 7f146f79d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:52.446 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.464+0000 7f146f79d640 1 -- 172.21.15.104:0/2158593875 >> v1:172.21.15.119:6789/0 conn(0x7f1468143d10 legacy=0x7f14681df540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:52.446 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.465+0000 7f146f79d640 1 -- 172.21.15.104:0/2158593875 shutdown_connections 2024-03-29T11:23:52.446 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.465+0000 7f146f79d640 1 -- 172.21.15.104:0/2158593875 >> 172.21.15.104:0/2158593875 conn(0x7f1468066a00 msgr2=0x7f14680b5a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:52.446 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.465+0000 7f146f79d640 1 -- 172.21.15.104:0/2158593875 shutdown_connections 2024-03-29T11:23:52.447 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.465+0000 7f146f79d640 1 -- 172.21.15.104:0/2158593875 wait complete. 2024-03-29T11:23:52.447 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:52.465+0000 7f146f79d640 1 librados: shutdown 2024-03-29T11:23:52.464 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:59.271 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:59.407 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.426+0000 7f62df59d640 10 monclient: get_monmap_and_config 2024-03-29T11:23:59.408 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.426+0000 7f62df59d640 10 monclient: build_initial_monmap 2024-03-29T11:23:59.408 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.426+0000 7f62df59d640 10 monclient: monmap: 2024-03-29T11:23:59.408 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:23:59.408 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:23:59.408 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:23:59.428181+0000 2024-03-29T11:23:59.408 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:23:59.428181+0000 2024-03-29T11:23:59.408 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:59.408 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:59.408 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:23:59.408 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:23:59.408 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:23:59.408 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:59.410 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.426+0000 7f62df59d640 1 Processor -- start 2024-03-29T11:23:59.410 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.427+0000 7f62df59d640 1 -- start start 2024-03-29T11:23:59.410 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.427+0000 7f62df59d640 10 monclient: init 2024-03-29T11:23:59.410 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.428+0000 7f62df59d640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:59.410 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.428+0000 7f62df59d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:23:59.410 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.428+0000 7f62df59d640 10 monclient(hunting): picked mon.noname-c con 0x7f62d81457f0 addr v1:172.21.15.119:6789/0 2024-03-29T11:23:59.410 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.428+0000 7f62df59d640 10 monclient(hunting): picked mon.noname-a con 0x7f62d813f650 addr v1:172.21.15.104:6789/0 2024-03-29T11:23:59.410 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.428+0000 7f62df59d640 10 monclient(hunting): picked mon.noname-b con 0x7f62d80a9d70 addr v1:172.21.15.104:6790/0 2024-03-29T11:23:59.410 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.428+0000 7f62df59d640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62d80aaef0 con 0x7f62d813f650 2024-03-29T11:23:59.410 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.428+0000 7f62dcb38640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f62d813f650 0x7f62d80b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:35232/0 (socket says 172.21.15.104:35232) 2024-03-29T11:23:59.410 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.428+0000 7f62dcb38640 1 -- 172.21.15.104:0/724802096 learned_addr learned my addr 172.21.15.104:0/724802096 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:23:59.421 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.429+0000 7f62cffff640 1 -- 172.21.15.104:0/724802096 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 199785722 0 0) 0x7f62d80aaef0 con 0x7f62d813f650 2024-03-29T11:23:59.421 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.430+0000 7f62df59d640 1 -- 172.21.15.104:0/724802096 --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62d8145bd0 con 0x7f62d81457f0 2024-03-29T11:23:59.421 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.430+0000 7f62df59d640 1 -- 172.21.15.104:0/724802096 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62d8145eb0 con 0x7f62d80a9d70 2024-03-29T11:23:59.421 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.430+0000 7f62df59d640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:59.422 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.430+0000 7f62df59d640 10 monclient(hunting): authenticate will time out at 1220.682154s 2024-03-29T11:23:59.422 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.430+0000 7f62cffff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:59.422 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.430+0000 7f62cffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:59.422 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.430+0000 7f62cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:59.422 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.430+0000 7f62cffff640 10 monclient(hunting): my global_id is 4395 2024-03-29T11:23:59.422 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.430+0000 7f62cffff640 1 -- 172.21.15.104:0/724802096 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62c4003520 con 0x7f62d813f650 2024-03-29T11:23:59.422 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.430+0000 7f62cffff640 1 -- 172.21.15.104:0/724802096 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2208837122 0 0) 0x7f62c4003520 con 0x7f62d813f650 2024-03-29T11:23:59.422 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.430+0000 7f62cffff640 1 -- 172.21.15.104:0/724802096 >> v1:172.21.15.104:6790/0 conn(0x7f62d80a9d70 legacy=0x7f62d814d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:23:59.422 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.431+0000 7f62cffff640 1 -- 172.21.15.104:0/724802096 >> v1:172.21.15.119:6789/0 conn(0x7f62d81457f0 legacy=0x7f62d813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:59.422 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.431+0000 7f62cffff640 10 monclient: _finish_hunting 0 2024-03-29T11:23:59.422 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.431+0000 7f62cffff640 1 monclient: found mon.noname-a 2024-03-29T11:23:59.422 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.431+0000 7f62cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:59.423 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.431+0000 7f62cffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.104:6789/0 2024-03-29T11:23:59.423 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.431+0000 7f62cffff640 1 -- 172.21.15.104:0/724802096 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d8146340 con 0x7f62d813f650 2024-03-29T11:23:59.423 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.431+0000 7f62cffff640 10 monclient: _finish_auth 0 2024-03-29T11:23:59.423 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.431+0000 7f62cffff640 10 monclient: _check_auth_tickets 2024-03-29T11:23:59.423 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.431+0000 7f62cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:59.423 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.431+0000 7f62cffff640 1 -- 172.21.15.104:0/724802096 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f62d4001a10 con 0x7f62d813f650 2024-03-29T11:23:59.423 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.431+0000 7f62cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:59.423 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.431+0000 7f62cffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:59.423 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.431+0000 7f62cffff640 10 monclient: dump: 2024-03-29T11:23:59.423 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:23:59.423 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:23:59.424 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:59.424 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:59.424 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:59.424 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:59.424 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:23:59.424 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:23:59.424 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:23:59.424 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:59.424 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.431+0000 7f62df59d640 5 monclient: authenticate success, global_id 4395 2024-03-29T11:23:59.424 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.431+0000 7f62df59d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:59.424 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.431+0000 7f62cffff640 1 -- 172.21.15.104:0/724802096 <== mon.0 v1:172.21.15.104:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62d4001be0 con 0x7f62d813f650 2024-03-29T11:23:59.424 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.431+0000 7f62cffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:59.424 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.431+0000 7f62cffff640 1 -- 172.21.15.104:0/724802096 <== mon.0 v1:172.21.15.104:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f62d4002030 con 0x7f62d813f650 2024-03-29T11:23:59.424 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.431+0000 7f62cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:59.425 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.431+0000 7f62cffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:59.425 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.431+0000 7f62cffff640 10 monclient: dump: 2024-03-29T11:23:59.425 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:23:59.425 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:23:59.425 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:59.425 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:59.425 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:59.425 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:59.425 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:23:59.425 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:23:59.425 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:23:59.426 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:59.426 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.431+0000 7f62df59d640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:59.426 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.431+0000 7f62df59d640 4 set_mon_vals no callback set 2024-03-29T11:23:59.426 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.432+0000 7f62df59d640 10 monclient: shutdown 2024-03-29T11:23:59.426 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.432+0000 7f62df59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:59.426 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.432+0000 7f62df59d640 1 -- 172.21.15.104:0/724802096 >> v1:172.21.15.104:6789/0 conn(0x7f62d813f650 legacy=0x7f62d80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:59.426 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.432+0000 7f62df59d640 1 -- 172.21.15.104:0/724802096 shutdown_connections 2024-03-29T11:23:59.426 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.432+0000 7f62df59d640 1 -- 172.21.15.104:0/724802096 >> 172.21.15.104:0/724802096 conn(0x7f62d8066a00 msgr2=0x7f62d8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:59.426 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.433+0000 7f62df59d640 1 -- 172.21.15.104:0/724802096 shutdown_connections 2024-03-29T11:23:59.426 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.433+0000 7f62df59d640 1 -- 172.21.15.104:0/724802096 wait complete. 2024-03-29T11:23:59.426 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.434+0000 7f62df59d640 10 monclient: build_initial_monmap 2024-03-29T11:23:59.426 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.434+0000 7f62df59d640 10 monclient: monmap: 2024-03-29T11:23:59.426 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:23:59.427 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:59.427 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:23:59.427 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:23:59.427 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:59.427 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:59.427 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:23:59.427 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:23:59.427 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:23:59.427 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:59.427 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.435+0000 7f62df59d640 1 librados: starting msgr at 2024-03-29T11:23:59.427 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.435+0000 7f62df59d640 1 librados: starting objecter 2024-03-29T11:23:59.428 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.435+0000 7f62df59d640 1 Processor -- start 2024-03-29T11:23:59.428 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.435+0000 7f62df59d640 1 -- start start 2024-03-29T11:23:59.428 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.435+0000 7f62df59d640 1 librados: setting wanted keys 2024-03-29T11:23:59.428 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.435+0000 7f62df59d640 1 librados: calling monclient init 2024-03-29T11:23:59.428 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.435+0000 7f62df59d640 10 monclient: init 2024-03-29T11:23:59.428 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.436+0000 7f62df59d640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:59.428 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.436+0000 7f62df59d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:23:59.428 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.436+0000 7f62df59d640 10 monclient(hunting): picked mon.noname-a con 0x7f62d80a9d70 addr v1:172.21.15.104:6789/0 2024-03-29T11:23:59.428 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.436+0000 7f62df59d640 10 monclient(hunting): picked mon.noname-b con 0x7f62d813f650 addr v1:172.21.15.119:6789/0 2024-03-29T11:23:59.428 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.436+0000 7f62df59d640 10 monclient(hunting): picked mon.noname-c con 0x7f62d81457f0 addr v1:172.21.15.104:6790/0 2024-03-29T11:23:59.428 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.436+0000 7f62df59d640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62d81cea70 con 0x7f62d80a9d70 2024-03-29T11:23:59.428 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.436+0000 7f62df59d640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62d81ced50 con 0x7f62d813f650 2024-03-29T11:23:59.429 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.436+0000 7f62df59d640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62d81cf030 con 0x7f62d81457f0 2024-03-29T11:23:59.429 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.436+0000 7f62df59d640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:59.429 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.436+0000 7f62df59d640 10 monclient(hunting): authenticate will time out at 1220.688324s 2024-03-29T11:23:59.429 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.436+0000 7f62dd339640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f62d80a9d70 0x7f62d81cd500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:38978/0 (socket says 172.21.15.104:38978) 2024-03-29T11:23:59.429 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.436+0000 7f62dd339640 1 -- 172.21.15.104:0/1545462774 learned_addr learned my addr 172.21.15.104:0/1545462774 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:23:59.429 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.437+0000 7f62cdffb640 10 client.?.objecter ms_handle_connect 0x7f62d80a9d70 2024-03-29T11:23:59.429 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.437+0000 7f62cdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:59.429 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.437+0000 7f62cdffb640 1 -- 172.21.15.104:0/1545462774 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3317377657 0 0) 0x7f62d81cea70 con 0x7f62d80a9d70 2024-03-29T11:23:59.429 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.437+0000 7f62cdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:59.429 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.437+0000 7f62cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:59.429 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.437+0000 7f62cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:59.429 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.437+0000 7f62cdffb640 10 monclient(hunting): my global_id is 4398 2024-03-29T11:23:59.430 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.437+0000 7f62cdffb640 1 -- 172.21.15.104:0/1545462774 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62bc004eb0 con 0x7f62d80a9d70 2024-03-29T11:23:59.430 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.437+0000 7f62cdffb640 10 client.?.objecter ms_handle_connect 0x7f62d813f650 2024-03-29T11:23:59.430 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.437+0000 7f62cdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:59.430 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.437+0000 7f62cdffb640 1 -- 172.21.15.104:0/1545462774 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2893722566 0 0) 0x7f62bc004eb0 con 0x7f62d80a9d70 2024-03-29T11:23:59.430 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.438+0000 7f62cdffb640 1 -- 172.21.15.104:0/1545462774 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62d81cea70 con 0x7f62d80a9d70 2024-03-29T11:23:59.430 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.438+0000 7f62cdffb640 1 -- 172.21.15.104:0/1545462774 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f62d0001990 con 0x7f62d80a9d70 2024-03-29T11:23:59.430 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.438+0000 7f62cdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:59.430 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.438+0000 7f62cdffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:59.430 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.438+0000 7f62cdffb640 10 monclient(hunting): dump: 2024-03-29T11:23:59.430 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:23:59.430 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:23:59.431 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:59.431 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:59.431 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:59.431 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:59.431 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:23:59.431 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:23:59.431 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:23:59.431 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:59.431 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.438+0000 7f62cdffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:59.431 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.438+0000 7f62cdffb640 1 -- 172.21.15.104:0/1545462774 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3078980740 0 0) 0x7f62d81ced50 con 0x7f62d813f650 2024-03-29T11:23:59.432 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.438+0000 7f62cdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:59.432 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.438+0000 7f62cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:59.432 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.438+0000 7f62cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:59.432 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.438+0000 7f62cdffb640 10 monclient(hunting): my global_id is 4378 2024-03-29T11:23:59.432 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.438+0000 7f62cdffb640 1 -- 172.21.15.104:0/1545462774 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62bc004eb0 con 0x7f62d813f650 2024-03-29T11:23:59.432 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.438+0000 7f62cdffb640 1 -- 172.21.15.104:0/1545462774 <== mon.0 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3099127022 0 0) 0x7f62d81cea70 con 0x7f62d80a9d70 2024-03-29T11:23:59.432 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.438+0000 7f62cdffb640 1 -- 172.21.15.104:0/1545462774 >> v1:172.21.15.104:6790/0 conn(0x7f62d81457f0 legacy=0x7f62d81e2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:59.432 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.438+0000 7f62cdffb640 1 -- 172.21.15.104:0/1545462774 >> v1:172.21.15.119:6789/0 conn(0x7f62d813f650 legacy=0x7f62d81cb7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:59.432 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.438+0000 7f62cdffb640 10 monclient: _finish_hunting 0 2024-03-29T11:23:59.432 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.438+0000 7f62cdffb640 1 monclient: found mon.a 2024-03-29T11:23:59.433 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.438+0000 7f62cdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:59.433 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.438+0000 7f62cdffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:23:59.433 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.438+0000 7f62cdffb640 1 -- 172.21.15.104:0/1545462774 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d81ccf60 con 0x7f62d80a9d70 2024-03-29T11:23:59.433 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.438+0000 7f62cdffb640 10 monclient: _finish_auth 0 2024-03-29T11:23:59.433 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.438+0000 7f62cdffb640 10 monclient: _check_auth_tickets 2024-03-29T11:23:59.433 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.438+0000 7f62cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:59.433 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.438+0000 7f62df59d640 5 monclient: authenticate success, global_id 4398 2024-03-29T11:23:59.433 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.438+0000 7f62df59d640 10 monclient: _renew_subs 2024-03-29T11:23:59.433 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.438+0000 7f62df59d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:23:59.433 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.438+0000 7f62df59d640 1 -- 172.21.15.104:0/1545462774 --> v1:172.21.15.104:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62d81cb340 con 0x7f62d80a9d70 2024-03-29T11:23:59.433 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.438+0000 7f62cdffb640 1 -- 172.21.15.104:0/1545462774 <== mon.0 v1:172.21.15.104:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62d0001c50 con 0x7f62d80a9d70 2024-03-29T11:23:59.433 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.438+0000 7f62cdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:59.433 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.438+0000 7f62cdffb640 1 -- 172.21.15.104:0/1545462774 <== mon.0 v1:172.21.15.104:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f62d0003a60 con 0x7f62d80a9d70 2024-03-29T11:23:59.434 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.438+0000 7f62cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:59.434 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.438+0000 7f62cdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:59.434 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.438+0000 7f62cdffb640 10 monclient: dump: 2024-03-29T11:23:59.434 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:23:59.434 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:23:59.434 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:59.434 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:23:59.434 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:59.434 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:23:59.434 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:23:59.434 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:23:59.434 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:23:59.434 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:23:59.434 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.438+0000 7f62df59d640 10 client.4398.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:59.434 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.438+0000 7f62df59d640 10 monclient: _renew_subs 2024-03-29T11:23:59.435 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.438+0000 7f62df59d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:23:59.435 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.438+0000 7f62df59d640 1 -- 172.21.15.104:0/1545462774 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f62d81ccbb0 con 0x7f62d80a9d70 2024-03-29T11:23:59.435 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.439+0000 7f62df59d640 1 librados: init done 2024-03-29T11:23:59.435 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.439+0000 7f62cf7fe640 4 set_mon_vals no callback set 2024-03-29T11:23:59.435 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.440+0000 7f62cdffb640 1 -- 172.21.15.104:0/1545462774 <== mon.0 v1:172.21.15.104:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f62d0003c40 con 0x7f62d80a9d70 2024-03-29T11:23:59.436 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.441+0000 7f62dcb38640 10 monclient: get_auth_request con 0x7f62bc06dc40 auth_method 0 2024-03-29T11:23:59.436 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.441+0000 7f62cdffb640 1 -- 172.21.15.104:0/1545462774 <== mon.0 v1:172.21.15.104:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f62d0084b20 con 0x7f62d80a9d70 2024-03-29T11:23:59.436 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.441+0000 7f62cdffb640 10 client.4398.objecter ms_dispatch 0x7f62d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:59.436 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.441+0000 7f62cdffb640 3 client.4398.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:59.436 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.441+0000 7f62cdffb640 3 client.4398.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:59.436 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.441+0000 7f62cdffb640 20 client.4398.objecter dump_active .. 0 homeless 2024-03-29T11:23:59.436 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.441+0000 7f62df59d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:59.436 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.441+0000 7f62df59d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:59.437 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.441+0000 7f62df59d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:23:59.437 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.441+0000 7f62df59d640 1 -- 172.21.15.104:0/1545462774 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62b8005080 con 0x7f62d80a9d70 2024-03-29T11:23:59.437 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.441+0000 7f62cdffb640 10 client.4398.objecter ms_handle_connect 0x7f62bc06dc40 2024-03-29T11:23:59.437 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.448+0000 7f62cdffb640 1 -- 172.21.15.104:0/1545462774 <== mon.0 v1:172.21.15.104:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f62d0050400 con 0x7f62d80a9d70 2024-03-29T11:23:59.437 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.448+0000 7f62cdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:59.437 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.448+0000 7f62cdffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:59.753 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.772+0000 7f62df59d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:59.753 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.772+0000 7f62df59d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:59.754 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.772+0000 7f62df59d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:23:59.754 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.772+0000 7f62df59d640 1 -- 172.21.15.104:0/1545462774 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62b8005330 con 0x7f62d80a9d70 2024-03-29T11:23:59.754 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.773+0000 7f62cdffb640 1 -- 172.21.15.104:0/1545462774 <== mon.0 v1:172.21.15.104:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f62d0052330 con 0x7f62d80a9d70 2024-03-29T11:23:59.754 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.773+0000 7f62cdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:59.754 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.773+0000 7f62cdffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:59.754 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:23:59.755 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:59.758 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.776+0000 7f62df59d640 10 librados: watch_flush enter 2024-03-29T11:23:59.758 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.776+0000 7f62df59d640 10 librados: watch_flush exit 2024-03-29T11:23:59.758 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.776+0000 7f62df59d640 20 client.4398.objecter shutdown clearing up homeless session... 2024-03-29T11:23:59.758 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.776+0000 7f62df59d640 10 client.4398.objecter successfully canceled tick 2024-03-29T11:23:59.758 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.776+0000 7f62df59d640 1 -- 172.21.15.104:0/1545462774 >> v1:172.21.15.104:6816/3341126660 conn(0x7f62bc06dc40 legacy=0x7f62bc070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:59.758 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.776+0000 7f62df59d640 10 monclient: shutdown 2024-03-29T11:23:59.758 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.776+0000 7f62df59d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:59.758 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.776+0000 7f62df59d640 1 -- 172.21.15.104:0/1545462774 >> v1:172.21.15.104:6789/0 conn(0x7f62d80a9d70 legacy=0x7f62d81cd500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:59.760 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.778+0000 7f62df59d640 1 -- 172.21.15.104:0/1545462774 shutdown_connections 2024-03-29T11:23:59.760 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.778+0000 7f62df59d640 1 -- 172.21.15.104:0/1545462774 >> 172.21.15.104:0/1545462774 conn(0x7f62d8066a00 msgr2=0x7f62d80b4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:59.760 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.779+0000 7f62df59d640 1 -- 172.21.15.104:0/1545462774 shutdown_connections 2024-03-29T11:23:59.760 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.779+0000 7f62df59d640 1 -- 172.21.15.104:0/1545462774 wait complete. 2024-03-29T11:23:59.762 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:23:59.781+0000 7f62df59d640 1 librados: shutdown 2024-03-29T11:23:59.777 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:06.585 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:06.744 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.763+0000 7f7c3976c640 10 monclient: get_monmap_and_config 2024-03-29T11:24:06.744 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.763+0000 7f7c3976c640 10 monclient: build_initial_monmap 2024-03-29T11:24:06.744 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.763+0000 7f7c3976c640 10 monclient: monmap: 2024-03-29T11:24:06.744 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:24:06.744 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:24:06.744 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:24:06.764326+0000 2024-03-29T11:24:06.744 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:24:06.764326+0000 2024-03-29T11:24:06.744 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:06.744 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:06.744 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:24:06.744 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:24:06.744 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:24:06.745 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:06.745 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.764+0000 7f7c3976c640 1 Processor -- start 2024-03-29T11:24:06.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.764+0000 7f7c3976c640 1 -- start start 2024-03-29T11:24:06.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.764+0000 7f7c3976c640 10 monclient: init 2024-03-29T11:24:06.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.765+0000 7f7c3976c640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:06.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.765+0000 7f7c3976c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:24:06.749 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.768+0000 7f7c3976c640 10 monclient(hunting): picked mon.noname-b con 0x7f7c3406c250 addr v1:172.21.15.104:6790/0 2024-03-29T11:24:06.749 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.768+0000 7f7c3976c640 10 monclient(hunting): picked mon.noname-c con 0x7f7c340ae650 addr v1:172.21.15.119:6789/0 2024-03-29T11:24:06.749 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.768+0000 7f7c3976c640 10 monclient(hunting): picked mon.noname-a con 0x7f7c3413f8c0 addr v1:172.21.15.104:6789/0 2024-03-29T11:24:06.749 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.768+0000 7f7c3976c640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c340adf50 con 0x7f7c3413f8c0 2024-03-29T11:24:06.749 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.768+0000 7f7c3976c640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c340ae230 con 0x7f7c340ae650 2024-03-29T11:24:06.749 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.768+0000 7f7c3976c640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c3413fd50 con 0x7f7c3406c250 2024-03-29T11:24:06.749 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.768+0000 7f7c3976c640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:06.749 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.768+0000 7f7c3976c640 10 monclient(hunting): authenticate will time out at 1228.019122s 2024-03-29T11:24:06.749 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.768+0000 7f7c32ffd640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f7c3406c250 0x7f7c341427b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:59869/0 (socket says 172.21.15.104:59869) 2024-03-29T11:24:06.749 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.768+0000 7f7c32ffd640 1 -- 172.21.15.104:0/1527936327 learned_addr learned my addr 172.21.15.104:0/1527936327 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:24:06.749 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.768+0000 7f7c31ffb640 1 -- 172.21.15.104:0/1527936327 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 646933101 0 0) 0x7f7c3413fd50 con 0x7f7c3406c250 2024-03-29T11:24:06.749 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.768+0000 7f7c31ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:24:06.750 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.768+0000 7f7c31ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:06.750 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.768+0000 7f7c31ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:06.750 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.768+0000 7f7c31ffb640 10 monclient(hunting): my global_id is 4409 2024-03-29T11:24:06.750 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.768+0000 7f7c31ffb640 1 -- 172.21.15.104:0/1527936327 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c1c003520 con 0x7f7c3406c250 2024-03-29T11:24:06.750 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.769+0000 7f7c31ffb640 1 -- 172.21.15.104:0/1527936327 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1486675419 0 0) 0x7f7c1c003520 con 0x7f7c3406c250 2024-03-29T11:24:06.750 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.769+0000 7f7c31ffb640 1 -- 172.21.15.104:0/1527936327 >> v1:172.21.15.119:6789/0 conn(0x7f7c340ae650 legacy=0x7f7c3413f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:24:06.750 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.769+0000 7f7c31ffb640 1 -- 172.21.15.104:0/1527936327 >> v1:172.21.15.104:6789/0 conn(0x7f7c3413f8c0 legacy=0x7f7c3414db30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:24:06.753 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.771+0000 7f7c31ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:24:06.753 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.772+0000 7f7c31ffb640 1 monclient: found mon.noname-b 2024-03-29T11:24:06.753 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.772+0000 7f7c31ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:06.753 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.772+0000 7f7c31ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.104:6790/0 2024-03-29T11:24:06.754 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.772+0000 7f7c31ffb640 1 -- 172.21.15.104:0/1527936327 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c3406db20 con 0x7f7c3406c250 2024-03-29T11:24:06.754 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.772+0000 7f7c31ffb640 10 monclient: _finish_auth 0 2024-03-29T11:24:06.754 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.772+0000 7f7c31ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:24:06.754 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.772+0000 7f7c31ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:06.754 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.772+0000 7f7c3976c640 5 monclient: authenticate success, global_id 4409 2024-03-29T11:24:06.754 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.772+0000 7f7c31ffb640 1 -- 172.21.15.104:0/1527936327 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f7c24001690 con 0x7f7c3406c250 2024-03-29T11:24:06.754 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.772+0000 7f7c3976c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:06.754 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.772+0000 7f7c31ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:06.754 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.772+0000 7f7c31ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:06.754 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.772+0000 7f7c31ffb640 10 monclient: dump: 2024-03-29T11:24:06.754 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:24:06.754 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:24:06.755 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:06.755 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:06.755 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:06.755 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:06.755 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:24:06.755 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:24:06.755 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:24:06.755 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:06.755 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.772+0000 7f7c3976c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:06.755 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.772+0000 7f7c31ffb640 1 -- 172.21.15.104:0/1527936327 <== mon.2 v1:172.21.15.104:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7c24001920 con 0x7f7c3406c250 2024-03-29T11:24:06.755 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.772+0000 7f7c31ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:06.755 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.772+0000 7f7c3976c640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:06.755 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.772+0000 7f7c3976c640 4 set_mon_vals no callback set 2024-03-29T11:24:06.755 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.772+0000 7f7c31ffb640 1 -- 172.21.15.104:0/1527936327 <== mon.2 v1:172.21.15.104:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f7c24001cf0 con 0x7f7c3406c250 2024-03-29T11:24:06.756 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.772+0000 7f7c31ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:06.756 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.772+0000 7f7c31ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:06.756 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.772+0000 7f7c31ffb640 10 monclient: dump: 2024-03-29T11:24:06.756 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:24:06.756 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:24:06.756 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:06.756 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:06.756 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:06.756 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:06.756 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:24:06.756 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:24:06.756 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:24:06.756 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:06.758 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.773+0000 7f7c3976c640 10 monclient: shutdown 2024-03-29T11:24:06.758 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.773+0000 7f7c3976c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:06.758 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.773+0000 7f7c3976c640 1 -- 172.21.15.104:0/1527936327 >> v1:172.21.15.104:6790/0 conn(0x7f7c3406c250 legacy=0x7f7c341427b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:06.758 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.775+0000 7f7c3976c640 1 -- 172.21.15.104:0/1527936327 shutdown_connections 2024-03-29T11:24:06.758 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.775+0000 7f7c3976c640 1 -- 172.21.15.104:0/1527936327 >> 172.21.15.104:0/1527936327 conn(0x7f7c34067610 msgr2=0x7f7c340b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:06.758 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.775+0000 7f7c3976c640 1 -- 172.21.15.104:0/1527936327 shutdown_connections 2024-03-29T11:24:06.758 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.775+0000 7f7c3976c640 1 -- 172.21.15.104:0/1527936327 wait complete. 2024-03-29T11:24:06.758 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.776+0000 7f7c3976c640 10 monclient: build_initial_monmap 2024-03-29T11:24:06.758 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.776+0000 7f7c3976c640 10 monclient: monmap: 2024-03-29T11:24:06.758 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:24:06.759 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:06.759 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:24:06.759 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:24:06.759 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:06.759 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:06.759 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:24:06.759 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:24:06.759 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:24:06.759 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:06.759 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.777+0000 7f7c3976c640 1 librados: starting msgr at 2024-03-29T11:24:06.759 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.777+0000 7f7c3976c640 1 librados: starting objecter 2024-03-29T11:24:06.759 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.777+0000 7f7c3976c640 1 Processor -- start 2024-03-29T11:24:06.759 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.777+0000 7f7c3976c640 1 -- start start 2024-03-29T11:24:06.759 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.777+0000 7f7c3976c640 1 librados: setting wanted keys 2024-03-29T11:24:06.759 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.777+0000 7f7c3976c640 1 librados: calling monclient init 2024-03-29T11:24:06.759 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.777+0000 7f7c3976c640 10 monclient: init 2024-03-29T11:24:06.760 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.778+0000 7f7c3976c640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:06.760 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.778+0000 7f7c3976c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:24:06.760 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.779+0000 7f7c3976c640 10 monclient(hunting): picked mon.noname-c con 0x7f7c3406c250 addr v1:172.21.15.104:6790/0 2024-03-29T11:24:06.760 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.779+0000 7f7c3976c640 10 monclient(hunting): picked mon.noname-b con 0x7f7c340ae650 addr v1:172.21.15.119:6789/0 2024-03-29T11:24:06.760 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.779+0000 7f7c3976c640 10 monclient(hunting): picked mon.noname-a con 0x7f7c3413f8c0 addr v1:172.21.15.104:6789/0 2024-03-29T11:24:06.761 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.779+0000 7f7c3976c640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c34072210 con 0x7f7c3413f8c0 2024-03-29T11:24:06.761 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.779+0000 7f7c3976c640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c341d7110 con 0x7f7c340ae650 2024-03-29T11:24:06.761 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.779+0000 7f7c3976c640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c34140010 con 0x7f7c3406c250 2024-03-29T11:24:06.761 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.779+0000 7f7c3976c640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:06.761 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.779+0000 7f7c3976c640 10 monclient(hunting): authenticate will time out at 1228.030911s 2024-03-29T11:24:06.761 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.780+0000 7f7c327fc640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f7c340ae650 0x7f7c34136ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:33372/0 (socket says 172.21.15.104:33372) 2024-03-29T11:24:06.761 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.780+0000 7f7c327fc640 1 -- 172.21.15.104:0/1462185980 learned_addr learned my addr 172.21.15.104:0/1462185980 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:24:06.761 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.780+0000 7f7c1b7fe640 10 client.?.objecter ms_handle_connect 0x7f7c340ae650 2024-03-29T11:24:06.762 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.780+0000 7f7c1b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:06.767 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.782+0000 7f7c1b7fe640 1 -- 172.21.15.104:0/1462185980 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1601368340 0 0) 0x7f7c341d7110 con 0x7f7c340ae650 2024-03-29T11:24:06.767 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.782+0000 7f7c1b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:06.767 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.782+0000 7f7c1b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:06.767 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.782+0000 7f7c1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:06.768 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.782+0000 7f7c1b7fe640 10 monclient(hunting): my global_id is 4381 2024-03-29T11:24:06.768 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.782+0000 7f7c1b7fe640 1 -- 172.21.15.104:0/1462185980 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c20004eb0 con 0x7f7c340ae650 2024-03-29T11:24:06.768 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.783+0000 7f7c1b7fe640 10 client.?.objecter ms_handle_connect 0x7f7c3406c250 2024-03-29T11:24:06.768 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.783+0000 7f7c1b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:06.768 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.783+0000 7f7c1b7fe640 1 -- 172.21.15.104:0/1462185980 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 565282381 0 0) 0x7f7c20004eb0 con 0x7f7c340ae650 2024-03-29T11:24:06.768 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.783+0000 7f7c1b7fe640 1 -- 172.21.15.104:0/1462185980 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c341d7110 con 0x7f7c340ae650 2024-03-29T11:24:06.768 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.783+0000 7f7c1b7fe640 1 -- 172.21.15.104:0/1462185980 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f7c28001ce0 con 0x7f7c340ae650 2024-03-29T11:24:06.768 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.783+0000 7f7c1b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:06.768 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.784+0000 7f7c1b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:06.768 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.784+0000 7f7c1b7fe640 10 monclient(hunting): dump: 2024-03-29T11:24:06.768 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:24:06.768 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:24:06.769 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:06.769 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:06.769 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:06.769 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:06.769 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:24:06.769 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:24:06.769 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:24:06.769 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:06.769 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.784+0000 7f7c1b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:06.769 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.784+0000 7f7c1b7fe640 10 client.?.objecter ms_handle_connect 0x7f7c3413f8c0 2024-03-29T11:24:06.769 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.784+0000 7f7c1b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:06.769 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.784+0000 7f7c1b7fe640 1 -- 172.21.15.104:0/1462185980 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1095130548 0 0) 0x7f7c34140010 con 0x7f7c3406c250 2024-03-29T11:24:06.769 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.784+0000 7f7c1b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:06.770 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.784+0000 7f7c1b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:06.770 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.784+0000 7f7c1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:06.770 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.784+0000 7f7c1b7fe640 10 monclient(hunting): my global_id is 4412 2024-03-29T11:24:06.770 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.784+0000 7f7c1b7fe640 1 -- 172.21.15.104:0/1462185980 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c20004eb0 con 0x7f7c3406c250 2024-03-29T11:24:06.770 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.784+0000 7f7c1b7fe640 1 -- 172.21.15.104:0/1462185980 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2529255586 0 0) 0x7f7c34072210 con 0x7f7c3413f8c0 2024-03-29T11:24:06.770 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.784+0000 7f7c1b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:06.770 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.784+0000 7f7c1b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:06.770 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.784+0000 7f7c1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:06.770 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.784+0000 7f7c1b7fe640 10 monclient(hunting): my global_id is 4404 2024-03-29T11:24:06.770 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.784+0000 7f7c1b7fe640 1 -- 172.21.15.104:0/1462185980 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c34140010 con 0x7f7c3413f8c0 2024-03-29T11:24:06.770 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.784+0000 7f7c1b7fe640 1 -- 172.21.15.104:0/1462185980 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 296188374 0 0) 0x7f7c20004eb0 con 0x7f7c3406c250 2024-03-29T11:24:06.771 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.784+0000 7f7c1b7fe640 1 -- 172.21.15.104:0/1462185980 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c34072210 con 0x7f7c3406c250 2024-03-29T11:24:06.771 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.784+0000 7f7c1b7fe640 1 -- 172.21.15.104:0/1462185980 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f7c24001550 con 0x7f7c3406c250 2024-03-29T11:24:06.771 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.784+0000 7f7c1b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:06.771 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.784+0000 7f7c1b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:06.773 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.784+0000 7f7c1b7fe640 10 monclient(hunting): dump: 2024-03-29T11:24:06.773 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:24:06.773 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:24:06.773 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:06.773 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:06.773 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:06.773 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:06.773 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:24:06.773 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:24:06.773 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:24:06.774 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:06.774 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.785+0000 7f7c1b7fe640 1 -- 172.21.15.104:0/1462185980 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1397047134 0 0) 0x7f7c341d7110 con 0x7f7c340ae650 2024-03-29T11:24:06.774 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.785+0000 7f7c1b7fe640 1 -- 172.21.15.104:0/1462185980 >> v1:172.21.15.104:6790/0 conn(0x7f7c3406c250 legacy=0x7f7c341d7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:06.774 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.785+0000 7f7c1b7fe640 1 -- 172.21.15.104:0/1462185980 >> v1:172.21.15.104:6789/0 conn(0x7f7c3413f8c0 legacy=0x7f7c34070850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:06.774 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.785+0000 7f7c1b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:24:06.774 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.785+0000 7f7c1b7fe640 1 monclient: found mon.b 2024-03-29T11:24:06.774 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.785+0000 7f7c1b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:06.774 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.785+0000 7f7c1b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:24:06.775 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.785+0000 7f7c1b7fe640 1 -- 172.21.15.104:0/1462185980 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c341401a0 con 0x7f7c340ae650 2024-03-29T11:24:06.775 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.785+0000 7f7c1b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:24:06.775 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.785+0000 7f7c1b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:24:06.775 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.785+0000 7f7c1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:06.775 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.785+0000 7f7c3976c640 5 monclient: authenticate success, global_id 4381 2024-03-29T11:24:06.775 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.785+0000 7f7c3976c640 10 monclient: _renew_subs 2024-03-29T11:24:06.775 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.785+0000 7f7c3976c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:24:06.775 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.785+0000 7f7c3976c640 1 -- 172.21.15.104:0/1462185980 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c341d7d40 con 0x7f7c340ae650 2024-03-29T11:24:06.775 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.785+0000 7f7c3976c640 10 client.4381.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:06.775 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.785+0000 7f7c3976c640 10 monclient: _renew_subs 2024-03-29T11:24:06.775 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.785+0000 7f7c3976c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:24:06.775 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.785+0000 7f7c3976c640 1 -- 172.21.15.104:0/1462185980 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c341373d0 con 0x7f7c340ae650 2024-03-29T11:24:06.775 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.785+0000 7f7c3976c640 1 librados: init done 2024-03-29T11:24:06.775 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.786+0000 7f7c1b7fe640 1 -- 172.21.15.104:0/1462185980 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7c28001fc0 con 0x7f7c340ae650 2024-03-29T11:24:06.776 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.786+0000 7f7c1b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:06.776 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.786+0000 7f7c1b7fe640 1 -- 172.21.15.104:0/1462185980 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f7c28003dc0 con 0x7f7c340ae650 2024-03-29T11:24:06.776 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.786+0000 7f7c1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:06.776 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.786+0000 7f7c1b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:06.776 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.786+0000 7f7c1b7fe640 10 monclient: dump: 2024-03-29T11:24:06.776 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:24:06.776 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:24:06.776 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:06.776 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:06.776 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:06.777 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:06.777 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:24:06.777 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:24:06.777 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:24:06.777 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:06.777 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.786+0000 7f7c317fa640 4 set_mon_vals no callback set 2024-03-29T11:24:06.777 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.786+0000 7f7c3976c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:06.777 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.788+0000 7f7c1b7fe640 1 -- 172.21.15.104:0/1462185980 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f7c28003f20 con 0x7f7c340ae650 2024-03-29T11:24:06.777 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.788+0000 7f7c1b7fe640 1 -- 172.21.15.104:0/1462185980 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f7c28084e00 con 0x7f7c340ae650 2024-03-29T11:24:06.777 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.788+0000 7f7c1b7fe640 10 client.4381.objecter ms_dispatch 0x7f7c34000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:06.777 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.788+0000 7f7c1b7fe640 3 client.4381.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:06.778 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.788+0000 7f7c1b7fe640 3 client.4381.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:06.778 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.788+0000 7f7c1b7fe640 20 client.4381.objecter dump_active .. 0 homeless 2024-03-29T11:24:06.778 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.788+0000 7f7c3976c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:06.778 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.788+0000 7f7c3976c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:24:06.778 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.788+0000 7f7c3976c640 1 -- 172.21.15.104:0/1462185980 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c14005080 con 0x7f7c340ae650 2024-03-29T11:24:06.778 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.789+0000 7f7c32ffd640 10 monclient: get_auth_request con 0x7f7c2006ef70 auth_method 0 2024-03-29T11:24:06.778 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.789+0000 7f7c1b7fe640 10 client.4381.objecter ms_handle_connect 0x7f7c2006ef70 2024-03-29T11:24:06.779 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.798+0000 7f7c1b7fe640 1 -- 172.21.15.104:0/1462185980 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7c280506e0 con 0x7f7c340ae650 2024-03-29T11:24:06.779 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.798+0000 7f7c1b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:06.779 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:06.798+0000 7f7c1b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:07.105 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:07.124+0000 7f7c3976c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:07.105 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:07.124+0000 7f7c3976c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:07.105 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:07.124+0000 7f7c3976c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:24:07.106 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:07.124+0000 7f7c3976c640 1 -- 172.21.15.104:0/1462185980 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7c14005330 con 0x7f7c340ae650 2024-03-29T11:24:07.107 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:07.126+0000 7f7c1b7fe640 1 -- 172.21.15.104:0/1462185980 <== mon.1 v1:172.21.15.119:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f7c28052610 con 0x7f7c340ae650 2024-03-29T11:24:07.107 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:07.126+0000 7f7c1b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:07.107 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:07.126+0000 7f7c1b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:07.107 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:24:07.108 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:07.111 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:07.130+0000 7f7c3976c640 10 librados: watch_flush enter 2024-03-29T11:24:07.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:07.130+0000 7f7c3976c640 10 librados: watch_flush exit 2024-03-29T11:24:07.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:07.130+0000 7f7c3976c640 20 client.4381.objecter shutdown clearing up homeless session... 2024-03-29T11:24:07.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:07.130+0000 7f7c3976c640 10 client.4381.objecter successfully canceled tick 2024-03-29T11:24:07.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:07.130+0000 7f7c3976c640 1 -- 172.21.15.104:0/1462185980 >> v1:172.21.15.104:6816/3341126660 conn(0x7f7c2006ef70 legacy=0x7f7c20071430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:07.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:07.130+0000 7f7c3976c640 10 monclient: shutdown 2024-03-29T11:24:07.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:07.130+0000 7f7c3976c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:07.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:07.130+0000 7f7c3976c640 1 -- 172.21.15.104:0/1462185980 >> v1:172.21.15.119:6789/0 conn(0x7f7c340ae650 legacy=0x7f7c34136ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:07.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:07.130+0000 7f7c3976c640 1 -- 172.21.15.104:0/1462185980 shutdown_connections 2024-03-29T11:24:07.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:07.130+0000 7f7c3976c640 1 -- 172.21.15.104:0/1462185980 >> 172.21.15.104:0/1462185980 conn(0x7f7c34067610 msgr2=0x7f7c3413ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:07.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:07.131+0000 7f7c3976c640 1 -- 172.21.15.104:0/1462185980 shutdown_connections 2024-03-29T11:24:07.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:07.131+0000 7f7c3976c640 1 -- 172.21.15.104:0/1462185980 wait complete. 2024-03-29T11:24:07.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:07.131+0000 7f7c3976c640 1 librados: shutdown 2024-03-29T11:24:07.129 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:13.936 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:14.085 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.103+0000 7fa041add640 10 monclient: get_monmap_and_config 2024-03-29T11:24:14.085 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.103+0000 7fa041add640 10 monclient: build_initial_monmap 2024-03-29T11:24:14.085 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.103+0000 7fa041add640 10 monclient: monmap: 2024-03-29T11:24:14.085 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:24:14.085 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:24:14.085 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:24:14.104395+0000 2024-03-29T11:24:14.085 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:24:14.104395+0000 2024-03-29T11:24:14.085 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:14.085 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:14.086 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:24:14.086 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:24:14.086 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:24:14.086 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:14.086 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.104+0000 7fa041add640 1 Processor -- start 2024-03-29T11:24:14.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.105+0000 7fa041add640 1 -- start start 2024-03-29T11:24:14.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.105+0000 7fa041add640 10 monclient: init 2024-03-29T11:24:14.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.105+0000 7fa041add640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:14.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.105+0000 7fa041add640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:24:14.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.105+0000 7fa041add640 10 monclient(hunting): picked mon.noname-c con 0x7fa03c0a9930 addr v1:172.21.15.119:6789/0 2024-03-29T11:24:14.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.105+0000 7fa041add640 10 monclient(hunting): picked mon.noname-a con 0x7fa03c149dc0 addr v1:172.21.15.104:6789/0 2024-03-29T11:24:14.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.105+0000 7fa041add640 10 monclient(hunting): picked mon.noname-b con 0x7fa03c14ff40 addr v1:172.21.15.104:6790/0 2024-03-29T11:24:14.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.105+0000 7fa041add640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa03c0aae60 con 0x7fa03c149dc0 2024-03-29T11:24:14.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.105+0000 7fa041add640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa03c0b4310 con 0x7fa03c0a9930 2024-03-29T11:24:14.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.105+0000 7fa041add640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa03c0afdd0 con 0x7fa03c14ff40 2024-03-29T11:24:14.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.105+0000 7fa041add640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:14.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.105+0000 7fa041add640 10 monclient(hunting): authenticate will time out at 1235.356873s 2024-03-29T11:24:14.089 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.108+0000 7fa03b7fe640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fa03c0a9930 0x7fa03c143a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:34780/0 (socket says 172.21.15.104:34780) 2024-03-29T11:24:14.089 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.108+0000 7fa03b7fe640 1 -- 172.21.15.104:0/1541050348 learned_addr learned my addr 172.21.15.104:0/1541050348 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:24:14.089 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.108+0000 7fa03a7fc640 1 -- 172.21.15.104:0/1541050348 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 544904425 0 0) 0x7fa03c0b4310 con 0x7fa03c0a9930 2024-03-29T11:24:14.089 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.108+0000 7fa03a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:14.089 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.108+0000 7fa03a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:14.089 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.108+0000 7fa03a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:14.089 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.108+0000 7fa03a7fc640 10 monclient(hunting): my global_id is 4384 2024-03-29T11:24:14.089 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.108+0000 7fa03a7fc640 1 -- 172.21.15.104:0/1541050348 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa024003520 con 0x7fa03c0a9930 2024-03-29T11:24:14.094 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.109+0000 7fa03a7fc640 1 -- 172.21.15.104:0/1541050348 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 773363897 0 0) 0x7fa03c0afdd0 con 0x7fa03c14ff40 2024-03-29T11:24:14.094 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.109+0000 7fa03a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:14.094 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.109+0000 7fa03a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:14.094 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.109+0000 7fa03a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:14.094 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.109+0000 7fa03a7fc640 10 monclient(hunting): my global_id is 4415 2024-03-29T11:24:14.094 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.109+0000 7fa03a7fc640 1 -- 172.21.15.104:0/1541050348 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa024003840 con 0x7fa03c14ff40 2024-03-29T11:24:14.094 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.109+0000 7fa03a7fc640 1 -- 172.21.15.104:0/1541050348 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3130392337 0 0) 0x7fa03c0aae60 con 0x7fa03c149dc0 2024-03-29T11:24:14.094 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.109+0000 7fa03a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:14.094 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.109+0000 7fa03a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:14.094 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.109+0000 7fa03a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:14.094 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.109+0000 7fa03a7fc640 10 monclient(hunting): my global_id is 4407 2024-03-29T11:24:14.095 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.109+0000 7fa03a7fc640 1 -- 172.21.15.104:0/1541050348 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa03c0afdd0 con 0x7fa03c149dc0 2024-03-29T11:24:14.095 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.109+0000 7fa03a7fc640 1 -- 172.21.15.104:0/1541050348 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 49362137 0 0) 0x7fa024003840 con 0x7fa03c14ff40 2024-03-29T11:24:14.095 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.109+0000 7fa03a7fc640 1 -- 172.21.15.104:0/1541050348 >> v1:172.21.15.119:6789/0 conn(0x7fa03c0a9930 legacy=0x7fa03c143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:14.095 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.109+0000 7fa03a7fc640 1 -- 172.21.15.104:0/1541050348 >> v1:172.21.15.104:6789/0 conn(0x7fa03c149dc0 legacy=0x7fa03c14a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:14.095 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.109+0000 7fa03a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:24:14.095 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.109+0000 7fa03a7fc640 1 monclient: found mon.noname-b 2024-03-29T11:24:14.095 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.109+0000 7fa03a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:14.095 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.109+0000 7fa03a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.104:6790/0 2024-03-29T11:24:14.095 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.109+0000 7fa03a7fc640 1 -- 172.21.15.104:0/1541050348 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa03c14a950 con 0x7fa03c14ff40 2024-03-29T11:24:14.095 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.109+0000 7fa03a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:24:14.095 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.109+0000 7fa03a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:24:14.095 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.109+0000 7fa03a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:14.095 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.109+0000 7fa03a7fc640 1 -- 172.21.15.104:0/1541050348 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fa0340019f0 con 0x7fa03c14ff40 2024-03-29T11:24:14.095 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.109+0000 7fa03a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:14.096 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.109+0000 7fa03a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:14.096 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.109+0000 7fa03a7fc640 10 monclient: dump: 2024-03-29T11:24:14.096 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:24:14.096 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:24:14.096 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:14.096 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:14.096 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:14.096 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:14.096 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:24:14.096 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:24:14.096 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:24:14.096 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:14.096 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.109+0000 7fa041add640 5 monclient: authenticate success, global_id 4415 2024-03-29T11:24:14.096 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.109+0000 7fa041add640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:14.097 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.109+0000 7fa03a7fc640 1 -- 172.21.15.104:0/1541050348 <== mon.2 v1:172.21.15.104:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa034001b50 con 0x7fa03c14ff40 2024-03-29T11:24:14.097 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.109+0000 7fa03a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:14.097 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.109+0000 7fa03a7fc640 1 -- 172.21.15.104:0/1541050348 <== mon.2 v1:172.21.15.104:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fa034001f60 con 0x7fa03c14ff40 2024-03-29T11:24:14.097 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.109+0000 7fa03a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:14.097 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.109+0000 7fa03a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:14.097 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.109+0000 7fa03a7fc640 10 monclient: dump: 2024-03-29T11:24:14.097 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:24:14.097 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:24:14.097 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:14.097 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:14.097 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:14.097 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:14.097 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:24:14.097 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:24:14.097 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:24:14.098 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:14.098 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.109+0000 7fa041add640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:14.098 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.109+0000 7fa041add640 4 set_mon_vals no callback set 2024-03-29T11:24:14.098 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.110+0000 7fa041add640 10 monclient: shutdown 2024-03-29T11:24:14.098 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.110+0000 7fa041add640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:14.098 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.110+0000 7fa041add640 1 -- 172.21.15.104:0/1541050348 >> v1:172.21.15.104:6790/0 conn(0x7fa03c14ff40 legacy=0x7fa03c152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:14.098 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.111+0000 7fa041add640 1 -- 172.21.15.104:0/1541050348 shutdown_connections 2024-03-29T11:24:14.098 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.111+0000 7fa041add640 1 -- 172.21.15.104:0/1541050348 >> 172.21.15.104:0/1541050348 conn(0x7fa03c0b35e0 msgr2=0x7fa03c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:14.098 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.112+0000 7fa041add640 1 -- 172.21.15.104:0/1541050348 shutdown_connections 2024-03-29T11:24:14.106 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.112+0000 7fa041add640 1 -- 172.21.15.104:0/1541050348 wait complete. 2024-03-29T11:24:14.106 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.113+0000 7fa041add640 10 monclient: build_initial_monmap 2024-03-29T11:24:14.106 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.113+0000 7fa041add640 10 monclient: monmap: 2024-03-29T11:24:14.106 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:24:14.106 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:14.106 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:24:14.106 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:24:14.107 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:14.107 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:14.107 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:24:14.107 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:24:14.107 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:24:14.107 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:14.107 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.113+0000 7fa041add640 1 librados: starting msgr at 2024-03-29T11:24:14.107 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.113+0000 7fa041add640 1 librados: starting objecter 2024-03-29T11:24:14.107 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.113+0000 7fa041add640 1 Processor -- start 2024-03-29T11:24:14.107 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.113+0000 7fa041add640 1 -- start start 2024-03-29T11:24:14.107 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.113+0000 7fa041add640 1 librados: setting wanted keys 2024-03-29T11:24:14.107 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.113+0000 7fa041add640 1 librados: calling monclient init 2024-03-29T11:24:14.107 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.113+0000 7fa041add640 10 monclient: init 2024-03-29T11:24:14.107 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.114+0000 7fa041add640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:14.108 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.114+0000 7fa041add640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:24:14.108 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.114+0000 7fa041add640 10 monclient(hunting): picked mon.noname-a con 0x7fa03c0a9930 addr v1:172.21.15.104:6789/0 2024-03-29T11:24:14.108 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.114+0000 7fa041add640 10 monclient(hunting): picked mon.noname-c con 0x7fa03c149dc0 addr v1:172.21.15.104:6790/0 2024-03-29T11:24:14.108 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.114+0000 7fa041add640 10 monclient(hunting): picked mon.noname-b con 0x7fa03c14ff40 addr v1:172.21.15.119:6789/0 2024-03-29T11:24:14.108 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.114+0000 7fa041add640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa03c13f440 con 0x7fa03c0a9930 2024-03-29T11:24:14.108 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.114+0000 7fa041add640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa03c071120 con 0x7fa03c14ff40 2024-03-29T11:24:14.108 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.114+0000 7fa041add640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa03c071400 con 0x7fa03c149dc0 2024-03-29T11:24:14.108 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.114+0000 7fa041add640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:14.108 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.114+0000 7fa041add640 10 monclient(hunting): authenticate will time out at 1235.365661s 2024-03-29T11:24:14.108 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.118+0000 7fa03affd640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7fa03c149dc0 0x7fa03c08b300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:44587/0 (socket says 172.21.15.104:44587) 2024-03-29T11:24:14.108 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.118+0000 7fa03affd640 1 -- 172.21.15.104:0/4053700384 learned_addr learned my addr 172.21.15.104:0/4053700384 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:24:14.108 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.118+0000 7fa023fff640 10 client.?.objecter ms_handle_connect 0x7fa03c0a9930 2024-03-29T11:24:14.108 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.118+0000 7fa023fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:14.108 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa023fff640 1 -- 172.21.15.104:0/4053700384 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3240393196 0 0) 0x7fa03c13f440 con 0x7fa03c0a9930 2024-03-29T11:24:14.109 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa023fff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:14.109 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa023fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:14.109 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa023fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:14.109 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa023fff640 10 monclient(hunting): my global_id is 4410 2024-03-29T11:24:14.109 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa023fff640 1 -- 172.21.15.104:0/4053700384 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa028004eb0 con 0x7fa03c0a9930 2024-03-29T11:24:14.109 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa023fff640 10 client.?.objecter ms_handle_connect 0x7fa03c14ff40 2024-03-29T11:24:14.109 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa023fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:14.109 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa023fff640 10 client.?.objecter ms_handle_connect 0x7fa03c149dc0 2024-03-29T11:24:14.109 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa023fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:14.109 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa023fff640 1 -- 172.21.15.104:0/4053700384 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1736731410 0 0) 0x7fa028004eb0 con 0x7fa03c0a9930 2024-03-29T11:24:14.109 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa023fff640 1 -- 172.21.15.104:0/4053700384 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa03c13f440 con 0x7fa03c0a9930 2024-03-29T11:24:14.109 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa023fff640 1 -- 172.21.15.104:0/4053700384 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2272262257 0 0) 0x7fa03c071120 con 0x7fa03c14ff40 2024-03-29T11:24:14.109 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa023fff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:14.109 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa023fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:14.110 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa023fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:14.110 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa023fff640 10 monclient(hunting): my global_id is 4387 2024-03-29T11:24:14.110 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa023fff640 1 -- 172.21.15.104:0/4053700384 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa028004eb0 con 0x7fa03c14ff40 2024-03-29T11:24:14.110 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa023fff640 1 -- 172.21.15.104:0/4053700384 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fa030001930 con 0x7fa03c0a9930 2024-03-29T11:24:14.110 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa023fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:14.110 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa023fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:14.110 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa023fff640 10 monclient(hunting): dump: 2024-03-29T11:24:14.110 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:24:14.110 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:24:14.110 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:14.110 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:14.110 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:14.110 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:14.110 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:24:14.110 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:24:14.111 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:24:14.111 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:14.111 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa023fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:14.111 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa023fff640 1 -- 172.21.15.104:0/4053700384 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2211195350 0 0) 0x7fa03c071400 con 0x7fa03c149dc0 2024-03-29T11:24:14.111 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa023fff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:14.111 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa023fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:14.111 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa023fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:14.111 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa023fff640 10 monclient(hunting): my global_id is 4418 2024-03-29T11:24:14.111 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa023fff640 1 -- 172.21.15.104:0/4053700384 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa03c071120 con 0x7fa03c149dc0 2024-03-29T11:24:14.111 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa023fff640 1 -- 172.21.15.104:0/4053700384 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3903311031 0 0) 0x7fa028004eb0 con 0x7fa03c14ff40 2024-03-29T11:24:14.111 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa023fff640 1 -- 172.21.15.104:0/4053700384 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa03c071400 con 0x7fa03c14ff40 2024-03-29T11:24:14.111 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa023fff640 1 -- 172.21.15.104:0/4053700384 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fa034001e60 con 0x7fa03c14ff40 2024-03-29T11:24:14.111 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa023fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:14.111 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa023fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:14.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa023fff640 10 monclient(hunting): dump: 2024-03-29T11:24:14.112 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:24:14.112 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:24:14.112 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:14.112 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:14.112 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:14.112 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:14.112 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:24:14.112 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:24:14.112 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:24:14.112 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:14.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa023fff640 1 -- 172.21.15.104:0/4053700384 <== mon.0 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 278635984 0 0) 0x7fa03c13f440 con 0x7fa03c0a9930 2024-03-29T11:24:14.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa023fff640 1 -- 172.21.15.104:0/4053700384 >> v1:172.21.15.104:6790/0 conn(0x7fa03c149dc0 legacy=0x7fa03c08b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:14.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa023fff640 1 -- 172.21.15.104:0/4053700384 >> v1:172.21.15.119:6789/0 conn(0x7fa03c14ff40 legacy=0x7fa03c06fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:14.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa023fff640 10 monclient: _finish_hunting 0 2024-03-29T11:24:14.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa023fff640 1 monclient: found mon.a 2024-03-29T11:24:14.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa023fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:14.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa023fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:24:14.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa023fff640 1 -- 172.21.15.104:0/4053700384 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa03c13f8d0 con 0x7fa03c0a9930 2024-03-29T11:24:14.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa023fff640 10 monclient: _finish_auth 0 2024-03-29T11:24:14.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa023fff640 10 monclient: _check_auth_tickets 2024-03-29T11:24:14.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa023fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:14.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa041add640 5 monclient: authenticate success, global_id 4410 2024-03-29T11:24:14.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa041add640 10 monclient: _renew_subs 2024-03-29T11:24:14.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa041add640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:24:14.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.119+0000 7fa041add640 1 -- 172.21.15.104:0/4053700384 --> v1:172.21.15.104:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa03c06f080 con 0x7fa03c0a9930 2024-03-29T11:24:14.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.120+0000 7fa041add640 10 client.4410.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:14.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.120+0000 7fa041add640 10 monclient: _renew_subs 2024-03-29T11:24:14.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.120+0000 7fa041add640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:24:14.114 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.120+0000 7fa041add640 1 -- 172.21.15.104:0/4053700384 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa03c0705d0 con 0x7fa03c0a9930 2024-03-29T11:24:14.114 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.120+0000 7fa041add640 1 librados: init done 2024-03-29T11:24:14.114 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.120+0000 7fa023fff640 1 -- 172.21.15.104:0/4053700384 <== mon.0 v1:172.21.15.104:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0300047e0 con 0x7fa03c0a9930 2024-03-29T11:24:14.114 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.120+0000 7fa023fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:14.114 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.120+0000 7fa023fff640 1 -- 172.21.15.104:0/4053700384 <== mon.0 v1:172.21.15.104:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fa030004af0 con 0x7fa03c0a9930 2024-03-29T11:24:14.114 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.120+0000 7fa023fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:14.114 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.120+0000 7fa023fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:14.114 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.120+0000 7fa023fff640 10 monclient: dump: 2024-03-29T11:24:14.114 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:24:14.114 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:24:14.114 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:14.114 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:14.114 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:14.114 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:14.115 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:24:14.115 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:24:14.115 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:24:14.115 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:14.115 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.120+0000 7fa039ffb640 4 set_mon_vals no callback set 2024-03-29T11:24:14.115 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.121+0000 7fa041add640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:14.115 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.121+0000 7fa041add640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:14.115 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.121+0000 7fa041add640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:24:14.115 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.121+0000 7fa041add640 1 -- 172.21.15.104:0/4053700384 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa03c075f10 con 0x7fa03c0a9930 2024-03-29T11:24:14.115 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.122+0000 7fa023fff640 1 -- 172.21.15.104:0/4053700384 <== mon.0 v1:172.21.15.104:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7fa03001aac0 con 0x7fa03c0a9930 2024-03-29T11:24:14.115 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.123+0000 7fa03affd640 10 monclient: get_auth_request con 0x7fa02806df50 auth_method 0 2024-03-29T11:24:14.115 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.123+0000 7fa023fff640 10 client.4410.objecter ms_handle_connect 0x7fa02806df50 2024-03-29T11:24:14.115 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.123+0000 7fa023fff640 1 -- 172.21.15.104:0/4053700384 <== mon.0 v1:172.21.15.104:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7fa030054850 con 0x7fa03c0a9930 2024-03-29T11:24:14.115 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.123+0000 7fa023fff640 10 client.4410.objecter ms_dispatch 0x7fa03c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:14.115 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.123+0000 7fa023fff640 3 client.4410.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:14.116 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.123+0000 7fa023fff640 3 client.4410.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:14.116 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.124+0000 7fa023fff640 20 client.4410.objecter dump_active .. 0 homeless 2024-03-29T11:24:14.116 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.132+0000 7fa023fff640 1 -- 172.21.15.104:0/4053700384 <== mon.0 v1:172.21.15.104:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa030053f20 con 0x7fa03c0a9930 2024-03-29T11:24:14.116 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.132+0000 7fa023fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:14.116 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.132+0000 7fa023fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:14.438 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.457+0000 7fa041add640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:14.438 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.457+0000 7fa041add640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:14.438 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.457+0000 7fa041add640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:24:14.438 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.457+0000 7fa041add640 1 -- 172.21.15.104:0/4053700384 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa03c06f210 con 0x7fa03c0a9930 2024-03-29T11:24:14.440 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.459+0000 7fa023fff640 1 -- 172.21.15.104:0/4053700384 <== mon.0 v1:172.21.15.104:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7fa030052290 con 0x7fa03c0a9930 2024-03-29T11:24:14.441 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.460+0000 7fa023fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:14.441 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.460+0000 7fa023fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:14.441 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:24:14.441 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:14.444 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.462+0000 7fa041add640 10 librados: watch_flush enter 2024-03-29T11:24:14.444 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.462+0000 7fa041add640 10 librados: watch_flush exit 2024-03-29T11:24:14.444 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.462+0000 7fa041add640 20 client.4410.objecter shutdown clearing up homeless session... 2024-03-29T11:24:14.444 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.462+0000 7fa041add640 10 client.4410.objecter successfully canceled tick 2024-03-29T11:24:14.444 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.462+0000 7fa041add640 1 -- 172.21.15.104:0/4053700384 >> v1:172.21.15.104:6816/3341126660 conn(0x7fa02806df50 legacy=0x7fa028070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:14.444 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.462+0000 7fa041add640 10 monclient: shutdown 2024-03-29T11:24:14.444 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.462+0000 7fa041add640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:14.444 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.462+0000 7fa041add640 1 -- 172.21.15.104:0/4053700384 >> v1:172.21.15.104:6789/0 conn(0x7fa03c0a9930 legacy=0x7fa03c071e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:14.444 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.462+0000 7fa041add640 1 -- 172.21.15.104:0/4053700384 shutdown_connections 2024-03-29T11:24:14.444 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.462+0000 7fa041add640 1 -- 172.21.15.104:0/4053700384 >> 172.21.15.104:0/4053700384 conn(0x7fa03c0b35e0 msgr2=0x7fa03c0b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:14.444 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.462+0000 7fa041add640 1 -- 172.21.15.104:0/4053700384 shutdown_connections 2024-03-29T11:24:14.444 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.462+0000 7fa041add640 1 -- 172.21.15.104:0/4053700384 wait complete. 2024-03-29T11:24:14.444 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:14.463+0000 7fa041add640 1 librados: shutdown 2024-03-29T11:24:14.458 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:21.266 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:21.415 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.431+0000 7f54c4e72640 10 monclient: get_monmap_and_config 2024-03-29T11:24:21.415 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.431+0000 7f54c4e72640 10 monclient: build_initial_monmap 2024-03-29T11:24:21.415 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.431+0000 7f54c4e72640 10 monclient: monmap: 2024-03-29T11:24:21.415 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:24:21.415 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:24:21.416 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:24:21.432321+0000 2024-03-29T11:24:21.416 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:24:21.432321+0000 2024-03-29T11:24:21.416 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:21.416 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:21.416 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:24:21.416 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:24:21.416 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:24:21.416 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:21.416 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.435+0000 7f54c4e72640 1 Processor -- start 2024-03-29T11:24:21.418 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.437+0000 7f54c4e72640 1 -- start start 2024-03-29T11:24:21.418 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.437+0000 7f54c4e72640 10 monclient: init 2024-03-29T11:24:21.419 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.437+0000 7f54c4e72640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:21.419 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.437+0000 7f54c4e72640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:21.419 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.437+0000 7f54c4e72640 10 monclient(hunting): picked mon.noname-a con 0x7f54c00a9750 addr v1:172.21.15.104:6789/0 2024-03-29T11:24:21.422 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.438+0000 7f54c4e72640 10 monclient(hunting): picked mon.noname-c con 0x7f54c00ae220 addr v1:172.21.15.119:6789/0 2024-03-29T11:24:21.422 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.440+0000 7f54c4e72640 10 monclient(hunting): picked mon.noname-b con 0x7f54c013b9d0 addr v1:172.21.15.104:6790/0 2024-03-29T11:24:21.422 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.440+0000 7f54c4e72640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54c0142490 con 0x7f54c00a9750 2024-03-29T11:24:21.422 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.440+0000 7f54c4e72640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54c00ac130 con 0x7f54c00ae220 2024-03-29T11:24:21.422 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.440+0000 7f54c4e72640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54c00ac410 con 0x7f54c013b9d0 2024-03-29T11:24:21.422 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.440+0000 7f54c4e72640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:21.422 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.441+0000 7f54c4e72640 10 monclient(hunting): authenticate will time out at 1242.692024s 2024-03-29T11:24:21.424 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.443+0000 7f54bf7fe640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f54c00a9750 0x7f54c0141d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:52551/0 (socket says 172.21.15.104:52551) 2024-03-29T11:24:21.424 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.443+0000 7f54bf7fe640 1 -- 172.21.15.104:0/1895678878 learned_addr learned my addr 172.21.15.104:0/1895678878 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:24:21.425 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.444+0000 7f54be7fc640 1 -- 172.21.15.104:0/1895678878 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4052327294 0 0) 0x7f54c00ac130 con 0x7f54c00ae220 2024-03-29T11:24:21.425 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.444+0000 7f54be7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:21.425 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.444+0000 7f54be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:21.425 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.444+0000 7f54be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:21.425 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.444+0000 7f54be7fc640 10 monclient(hunting): my global_id is 4390 2024-03-29T11:24:21.425 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.444+0000 7f54be7fc640 1 -- 172.21.15.104:0/1895678878 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54ac003520 con 0x7f54c00ae220 2024-03-29T11:24:21.426 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.444+0000 7f54be7fc640 1 -- 172.21.15.104:0/1895678878 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 35097864 0 0) 0x7f54c00ac410 con 0x7f54c013b9d0 2024-03-29T11:24:21.426 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.444+0000 7f54be7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:21.426 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.444+0000 7f54be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:21.426 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.444+0000 7f54be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:21.427 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.444+0000 7f54be7fc640 10 monclient(hunting): my global_id is 4421 2024-03-29T11:24:21.427 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.444+0000 7f54be7fc640 1 -- 172.21.15.104:0/1895678878 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54ac004a20 con 0x7f54c013b9d0 2024-03-29T11:24:21.427 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.444+0000 7f54be7fc640 1 -- 172.21.15.104:0/1895678878 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1299349943 0 0) 0x7f54c0142490 con 0x7f54c00a9750 2024-03-29T11:24:21.427 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.444+0000 7f54be7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:21.427 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.444+0000 7f54be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:21.427 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.444+0000 7f54be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:21.427 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.444+0000 7f54be7fc640 10 monclient(hunting): my global_id is 4413 2024-03-29T11:24:21.427 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.444+0000 7f54be7fc640 1 -- 172.21.15.104:0/1895678878 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54ac003760 con 0x7f54c00a9750 2024-03-29T11:24:21.427 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.444+0000 7f54be7fc640 1 -- 172.21.15.104:0/1895678878 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3999594875 0 0) 0x7f54ac003520 con 0x7f54c00ae220 2024-03-29T11:24:21.427 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.444+0000 7f54be7fc640 1 -- 172.21.15.104:0/1895678878 >> v1:172.21.15.104:6790/0 conn(0x7f54c013b9d0 legacy=0x7f54c0149930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:21.427 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.444+0000 7f54be7fc640 1 -- 172.21.15.104:0/1895678878 >> v1:172.21.15.104:6789/0 conn(0x7f54c00a9750 legacy=0x7f54c0141d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:21.427 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.444+0000 7f54be7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:24:21.427 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.444+0000 7f54be7fc640 1 monclient: found mon.noname-c 2024-03-29T11:24:21.427 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.444+0000 7f54be7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:21.428 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.444+0000 7f54be7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.119:6789/0 2024-03-29T11:24:21.428 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.444+0000 7f54be7fc640 1 -- 172.21.15.104:0/1895678878 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c00afa40 con 0x7f54c00ae220 2024-03-29T11:24:21.428 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.444+0000 7f54be7fc640 10 monclient: _finish_auth 0 2024-03-29T11:24:21.428 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.444+0000 7f54be7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:24:21.428 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.444+0000 7f54be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:21.428 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.445+0000 7f54be7fc640 1 -- 172.21.15.104:0/1895678878 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f54b8001670 con 0x7f54c00ae220 2024-03-29T11:24:21.428 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.445+0000 7f54be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:21.428 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.445+0000 7f54be7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:21.428 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.445+0000 7f54be7fc640 10 monclient: dump: 2024-03-29T11:24:21.428 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:24:21.428 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:24:21.428 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:21.428 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:21.428 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:21.429 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:21.429 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:24:21.429 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:24:21.429 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:24:21.429 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:21.429 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.445+0000 7f54c4e72640 5 monclient: authenticate success, global_id 4390 2024-03-29T11:24:21.429 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.445+0000 7f54c4e72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:21.429 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.445+0000 7f54be7fc640 1 -- 172.21.15.104:0/1895678878 <== mon.1 v1:172.21.15.119:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54b8001850 con 0x7f54c00ae220 2024-03-29T11:24:21.429 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.445+0000 7f54be7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:21.429 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.445+0000 7f54be7fc640 1 -- 172.21.15.104:0/1895678878 <== mon.1 v1:172.21.15.119:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f54b8001670 con 0x7f54c00ae220 2024-03-29T11:24:21.429 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.445+0000 7f54be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:21.429 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.445+0000 7f54be7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:21.429 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.445+0000 7f54be7fc640 10 monclient: dump: 2024-03-29T11:24:21.429 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:24:21.429 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:24:21.430 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:21.430 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:21.430 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:21.430 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:21.430 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:24:21.430 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:24:21.430 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:24:21.430 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:21.430 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.445+0000 7f54c4e72640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:21.430 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.445+0000 7f54c4e72640 4 set_mon_vals no callback set 2024-03-29T11:24:21.431 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.446+0000 7f54c4e72640 10 monclient: shutdown 2024-03-29T11:24:21.431 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.446+0000 7f54c4e72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:21.431 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.446+0000 7f54c4e72640 1 -- 172.21.15.104:0/1895678878 >> v1:172.21.15.119:6789/0 conn(0x7f54c00ae220 legacy=0x7f54c013b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:21.431 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.448+0000 7f54c4e72640 1 -- 172.21.15.104:0/1895678878 shutdown_connections 2024-03-29T11:24:21.431 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.448+0000 7f54c4e72640 1 -- 172.21.15.104:0/1895678878 >> 172.21.15.104:0/1895678878 conn(0x7f54c00673e0 msgr2=0x7f54c00b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:21.436 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.452+0000 7f54c4e72640 1 -- 172.21.15.104:0/1895678878 shutdown_connections 2024-03-29T11:24:21.436 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.452+0000 7f54c4e72640 1 -- 172.21.15.104:0/1895678878 wait complete. 2024-03-29T11:24:21.436 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.454+0000 7f54c4e72640 10 monclient: build_initial_monmap 2024-03-29T11:24:21.436 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.454+0000 7f54c4e72640 10 monclient: monmap: 2024-03-29T11:24:21.436 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:24:21.436 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:21.436 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:24:21.437 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:24:21.437 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:21.437 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:21.437 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:24:21.437 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:24:21.437 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:24:21.437 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:21.437 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.454+0000 7f54c4e72640 1 librados: starting msgr at 2024-03-29T11:24:21.437 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.454+0000 7f54c4e72640 1 librados: starting objecter 2024-03-29T11:24:21.437 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.454+0000 7f54c4e72640 1 Processor -- start 2024-03-29T11:24:21.437 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.454+0000 7f54c4e72640 1 -- start start 2024-03-29T11:24:21.437 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.455+0000 7f54c4e72640 1 librados: setting wanted keys 2024-03-29T11:24:21.437 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.455+0000 7f54c4e72640 1 librados: calling monclient init 2024-03-29T11:24:21.438 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.455+0000 7f54c4e72640 10 monclient: init 2024-03-29T11:24:21.438 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.456+0000 7f54c4e72640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:21.438 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.456+0000 7f54c4e72640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:24:21.440 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.456+0000 7f54c4e72640 10 monclient(hunting): picked mon.noname-c con 0x7f54c00a9750 addr v1:172.21.15.104:6790/0 2024-03-29T11:24:21.442 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.459+0000 7f54c4e72640 10 monclient(hunting): picked mon.noname-b con 0x7f54c00ae220 addr v1:172.21.15.119:6789/0 2024-03-29T11:24:21.442 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.460+0000 7f54c4e72640 10 monclient(hunting): picked mon.noname-a con 0x7f54c013b9d0 addr v1:172.21.15.104:6789/0 2024-03-29T11:24:21.442 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.460+0000 7f54c4e72640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54c0140f20 con 0x7f54c013b9d0 2024-03-29T11:24:21.442 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.460+0000 7f54c4e72640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54c013f1f0 con 0x7f54c00ae220 2024-03-29T11:24:21.442 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.461+0000 7f54c4e72640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54c013a790 con 0x7f54c00a9750 2024-03-29T11:24:21.443 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.461+0000 7f54c4e72640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:21.443 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.461+0000 7f54c4e72640 10 monclient(hunting): authenticate will time out at 1242.712028s 2024-03-29T11:24:21.445 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.463+0000 7f54beffd640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f54c00ae220 0x7f54c012f990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:60983/0 (socket says 172.21.15.104:60983) 2024-03-29T11:24:21.445 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.463+0000 7f54beffd640 1 -- 172.21.15.104:0/3937316487 learned_addr learned my addr 172.21.15.104:0/3937316487 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:24:21.445 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.460+0000 7f54bf7fe640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f54c00a9750 0x7f54c00b2e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:56997/0 (socket says 172.21.15.104:56997) 2024-03-29T11:24:21.446 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.465+0000 7f54b7fff640 10 client.?.objecter ms_handle_connect 0x7f54c00ae220 2024-03-29T11:24:21.446 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.465+0000 7f54b7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:21.447 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.465+0000 7f54b7fff640 1 -- 172.21.15.104:0/3937316487 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2997988912 0 0) 0x7f54c013f1f0 con 0x7f54c00ae220 2024-03-29T11:24:21.447 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.465+0000 7f54b7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:21.447 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.465+0000 7f54b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:21.447 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.465+0000 7f54b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:21.447 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.465+0000 7f54b7fff640 10 monclient(hunting): my global_id is 4393 2024-03-29T11:24:21.447 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.465+0000 7f54b7fff640 1 -- 172.21.15.104:0/3937316487 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54a0004eb0 con 0x7f54c00ae220 2024-03-29T11:24:21.450 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.466+0000 7f54b7fff640 10 client.?.objecter ms_handle_connect 0x7f54c00a9750 2024-03-29T11:24:21.450 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.466+0000 7f54b7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:21.450 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.466+0000 7f54b7fff640 10 client.?.objecter ms_handle_connect 0x7f54c013b9d0 2024-03-29T11:24:21.450 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.466+0000 7f54b7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:21.450 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.466+0000 7f54b7fff640 1 -- 172.21.15.104:0/3937316487 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2574692820 0 0) 0x7f54c0140f20 con 0x7f54c013b9d0 2024-03-29T11:24:21.450 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.466+0000 7f54b7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:21.450 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.466+0000 7f54b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:21.450 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.466+0000 7f54b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:21.450 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.466+0000 7f54b7fff640 10 monclient(hunting): my global_id is 4416 2024-03-29T11:24:21.450 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.466+0000 7f54b7fff640 1 -- 172.21.15.104:0/3937316487 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54c013f1f0 con 0x7f54c013b9d0 2024-03-29T11:24:21.451 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.467+0000 7f54b7fff640 1 -- 172.21.15.104:0/3937316487 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3696973963 0 0) 0x7f54c013f1f0 con 0x7f54c013b9d0 2024-03-29T11:24:21.451 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.467+0000 7f54b7fff640 1 -- 172.21.15.104:0/3937316487 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54c0140f20 con 0x7f54c013b9d0 2024-03-29T11:24:21.451 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.467+0000 7f54b7fff640 1 -- 172.21.15.104:0/3937316487 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f54b0001a90 con 0x7f54c013b9d0 2024-03-29T11:24:21.451 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.467+0000 7f54b7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:21.451 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.467+0000 7f54b7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:21.451 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.467+0000 7f54b7fff640 10 monclient(hunting): dump: 2024-03-29T11:24:21.451 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:24:21.451 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:24:21.451 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:21.451 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:21.451 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:21.451 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:21.451 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:24:21.451 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:24:21.451 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:24:21.452 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:21.452 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.467+0000 7f54b7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:21.452 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.467+0000 7f54b7fff640 1 -- 172.21.15.104:0/3937316487 <== mon.0 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 237977992 0 0) 0x7f54c0140f20 con 0x7f54c013b9d0 2024-03-29T11:24:21.452 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.467+0000 7f54b7fff640 1 -- 172.21.15.104:0/3937316487 >> v1:172.21.15.104:6790/0 conn(0x7f54c00a9750 legacy=0x7f54c00b2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:21.452 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.467+0000 7f54b7fff640 1 -- 172.21.15.104:0/3937316487 >> v1:172.21.15.119:6789/0 conn(0x7f54c00ae220 legacy=0x7f54c012f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:21.452 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.467+0000 7f54b7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:24:21.452 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.467+0000 7f54b7fff640 1 monclient: found mon.a 2024-03-29T11:24:21.452 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.467+0000 7f54b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:21.452 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.467+0000 7f54b7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:24:21.452 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.467+0000 7f54b7fff640 1 -- 172.21.15.104:0/3937316487 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c013a270 con 0x7f54c013b9d0 2024-03-29T11:24:21.452 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.467+0000 7f54b7fff640 10 monclient: _finish_auth 0 2024-03-29T11:24:21.452 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.468+0000 7f54b7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:24:21.452 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.468+0000 7f54b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:21.453 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.468+0000 7f54b7fff640 1 -- 172.21.15.104:0/3937316487 <== mon.0 v1:172.21.15.104:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54b0001fa0 con 0x7f54c013b9d0 2024-03-29T11:24:21.453 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.468+0000 7f54c4e72640 5 monclient: authenticate success, global_id 4416 2024-03-29T11:24:21.453 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.468+0000 7f54c4e72640 10 monclient: _renew_subs 2024-03-29T11:24:21.453 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.468+0000 7f54c4e72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:24:21.455 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.468+0000 7f54c4e72640 1 -- 172.21.15.104:0/3937316487 --> v1:172.21.15.104:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54c01300a0 con 0x7f54c013b9d0 2024-03-29T11:24:21.455 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.470+0000 7f54c4e72640 10 client.4416.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:21.455 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.470+0000 7f54c4e72640 10 monclient: _renew_subs 2024-03-29T11:24:21.455 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.470+0000 7f54c4e72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:24:21.455 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.470+0000 7f54c4e72640 1 -- 172.21.15.104:0/3937316487 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f54c013f6f0 con 0x7f54c013b9d0 2024-03-29T11:24:21.455 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.470+0000 7f54c4e72640 1 librados: init done 2024-03-29T11:24:21.455 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.471+0000 7f54b7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:21.455 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.471+0000 7f54b5ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:21.455 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.471+0000 7f54b7fff640 1 -- 172.21.15.104:0/3937316487 <== mon.0 v1:172.21.15.104:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f54b0003b10 con 0x7f54c013b9d0 2024-03-29T11:24:21.455 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.471+0000 7f54b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:21.455 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.471+0000 7f54b7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:21.455 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.471+0000 7f54b7fff640 10 monclient: dump: 2024-03-29T11:24:21.455 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:24:21.456 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:24:21.456 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:21.456 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:21.456 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:21.456 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:21.456 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:24:21.456 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:24:21.456 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:24:21.456 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:21.456 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.472+0000 7f54bdffb640 4 set_mon_vals no callback set 2024-03-29T11:24:21.456 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.472+0000 7f54b5ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:21.456 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.473+0000 7f54b5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:24:21.456 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.473+0000 7f54b5ffb640 1 -- 172.21.15.104:0/3937316487 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54a4005080 con 0x7f54c013b9d0 2024-03-29T11:24:21.458 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.476+0000 7f54b7fff640 1 -- 172.21.15.104:0/3937316487 <== mon.0 v1:172.21.15.104:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f54b0019ae0 con 0x7f54c013b9d0 2024-03-29T11:24:21.458 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.477+0000 7f54b7fff640 1 -- 172.21.15.104:0/3937316487 <== mon.0 v1:172.21.15.104:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f54b0083a00 con 0x7f54c013b9d0 2024-03-29T11:24:21.458 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.477+0000 7f54b7fff640 10 client.4416.objecter ms_dispatch 0x7f54c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:21.458 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.477+0000 7f54b7fff640 3 client.4416.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:21.459 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.477+0000 7f54b7fff640 3 client.4416.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:21.459 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.477+0000 7f54b7fff640 20 client.4416.objecter dump_active .. 0 homeless 2024-03-29T11:24:21.459 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.478+0000 7f54bf7fe640 10 monclient: get_auth_request con 0x7f54a006dd60 auth_method 0 2024-03-29T11:24:21.459 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.478+0000 7f54b7fff640 10 client.4416.objecter ms_handle_connect 0x7f54a006dd60 2024-03-29T11:24:21.464 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.483+0000 7f54b7fff640 1 -- 172.21.15.104:0/3937316487 <== mon.0 v1:172.21.15.104:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f54b004f2e0 con 0x7f54c013b9d0 2024-03-29T11:24:21.464 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.483+0000 7f54b7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:21.464 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.483+0000 7f54b7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:21.784 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.803+0000 7f54b5ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:21.785 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.803+0000 7f54b5ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:21.785 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.803+0000 7f54b5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:24:21.785 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.803+0000 7f54b5ffb640 1 -- 172.21.15.104:0/3937316487 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54a4005330 con 0x7f54c013b9d0 2024-03-29T11:24:21.786 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.805+0000 7f54b7fff640 1 -- 172.21.15.104:0/3937316487 <== mon.0 v1:172.21.15.104:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f54b0051210 con 0x7f54c013b9d0 2024-03-29T11:24:21.786 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.805+0000 7f54b7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:21.786 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.805+0000 7f54b7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:21.788 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:24:21.788 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:21.791 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.810+0000 7f54b5ffb640 10 librados: watch_flush enter 2024-03-29T11:24:21.792 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.810+0000 7f54b5ffb640 10 librados: watch_flush exit 2024-03-29T11:24:21.792 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.810+0000 7f54b5ffb640 20 client.4416.objecter shutdown clearing up homeless session... 2024-03-29T11:24:21.792 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.810+0000 7f54b5ffb640 10 client.4416.objecter successfully canceled tick 2024-03-29T11:24:21.792 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.810+0000 7f54b5ffb640 1 -- 172.21.15.104:0/3937316487 >> v1:172.21.15.104:6816/3341126660 conn(0x7f54a006dd60 legacy=0x7f54a0070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:21.792 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.810+0000 7f54b5ffb640 10 monclient: shutdown 2024-03-29T11:24:21.792 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.811+0000 7f54b5ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:21.792 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.811+0000 7f54b5ffb640 1 -- 172.21.15.104:0/3937316487 >> v1:172.21.15.104:6789/0 conn(0x7f54c013b9d0 legacy=0x7f54c01d2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:21.792 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.811+0000 7f54b5ffb640 1 -- 172.21.15.104:0/3937316487 shutdown_connections 2024-03-29T11:24:21.793 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.811+0000 7f54b5ffb640 1 -- 172.21.15.104:0/3937316487 >> 172.21.15.104:0/3937316487 conn(0x7f54c00673e0 msgr2=0x7f54c0069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:21.793 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.811+0000 7f54b5ffb640 1 -- 172.21.15.104:0/3937316487 shutdown_connections 2024-03-29T11:24:21.793 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.811+0000 7f54b5ffb640 1 -- 172.21.15.104:0/3937316487 wait complete. 2024-03-29T11:24:21.793 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:21.811+0000 7f54b5ffb640 1 librados: shutdown 2024-03-29T11:24:21.809 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:28.616 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:28.768 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.786+0000 7f4457fff640 10 monclient: get_monmap_and_config 2024-03-29T11:24:28.768 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.786+0000 7f4457fff640 10 monclient: build_initial_monmap 2024-03-29T11:24:28.769 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.786+0000 7f4457fff640 10 monclient: monmap: 2024-03-29T11:24:28.769 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:24:28.769 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:24:28.769 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:24:28.787244+0000 2024-03-29T11:24:28.769 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:24:28.787244+0000 2024-03-29T11:24:28.769 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:28.769 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:28.769 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:24:28.769 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:24:28.769 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:24:28.769 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:28.769 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.787+0000 7f4457fff640 1 Processor -- start 2024-03-29T11:24:28.773 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.788+0000 7f4457fff640 1 -- start start 2024-03-29T11:24:28.773 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.788+0000 7f4457fff640 10 monclient: init 2024-03-29T11:24:28.773 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.789+0000 7f4457fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:28.773 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.789+0000 7f4457fff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:24:28.773 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.789+0000 7f4457fff640 10 monclient(hunting): picked mon.noname-b con 0x7f44580b52b0 addr v1:172.21.15.104:6790/0 2024-03-29T11:24:28.773 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.789+0000 7f4457fff640 10 monclient(hunting): picked mon.noname-c con 0x7f44580aec00 addr v1:172.21.15.119:6789/0 2024-03-29T11:24:28.773 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.789+0000 7f4457fff640 10 monclient(hunting): picked mon.noname-a con 0x7f44580a9f00 addr v1:172.21.15.104:6789/0 2024-03-29T11:24:28.774 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.789+0000 7f4457fff640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44580aa6e0 con 0x7f44580a9f00 2024-03-29T11:24:28.774 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.789+0000 7f4457fff640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f445813c8a0 con 0x7f44580aec00 2024-03-29T11:24:28.774 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.789+0000 7f4457fff640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44580aa390 con 0x7f44580b52b0 2024-03-29T11:24:28.774 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.789+0000 7f4457fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:28.774 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.789+0000 7f4457fff640 10 monclient(hunting): authenticate will time out at 1250.040548s 2024-03-29T11:24:28.776 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.794+0000 7f44577fe640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f44580a9f00 0x7f445813f530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:58371/0 (socket says 172.21.15.104:58371) 2024-03-29T11:24:28.776 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.794+0000 7f44577fe640 1 -- 172.21.15.104:0/3667791890 learned_addr learned my addr 172.21.15.104:0/3667791890 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:24:28.776 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.794+0000 7f4455ffb640 1 -- 172.21.15.104:0/3667791890 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 400812390 0 0) 0x7f44580aa6e0 con 0x7f44580a9f00 2024-03-29T11:24:28.776 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.794+0000 7f4455ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:24:28.776 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.794+0000 7f4455ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:28.777 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.794+0000 7f4455ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:28.777 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.794+0000 7f4455ffb640 10 monclient(hunting): my global_id is 4419 2024-03-29T11:24:28.777 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.794+0000 7f4455ffb640 1 -- 172.21.15.104:0/3667791890 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4444003520 con 0x7f44580a9f00 2024-03-29T11:24:28.777 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.795+0000 7f4455ffb640 1 -- 172.21.15.104:0/3667791890 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 566020554 0 0) 0x7f4444003520 con 0x7f44580a9f00 2024-03-29T11:24:28.778 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.796+0000 7f4455ffb640 1 -- 172.21.15.104:0/3667791890 >> v1:172.21.15.104:6790/0 conn(0x7f44580b52b0 legacy=0x7f44580ae4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:24:28.778 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.796+0000 7f4455ffb640 1 -- 172.21.15.104:0/3667791890 >> v1:172.21.15.119:6789/0 conn(0x7f44580aec00 legacy=0x7f445813ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:28.778 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.796+0000 7f4455ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:24:28.778 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.796+0000 7f4455ffb640 1 monclient: found mon.noname-a 2024-03-29T11:24:28.778 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.796+0000 7f4455ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:28.778 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.796+0000 7f4455ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.104:6789/0 2024-03-29T11:24:28.778 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.796+0000 7f4455ffb640 1 -- 172.21.15.104:0/3667791890 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44580ab810 con 0x7f44580a9f00 2024-03-29T11:24:28.778 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.796+0000 7f4455ffb640 10 monclient: _finish_auth 0 2024-03-29T11:24:28.778 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.796+0000 7f4455ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:24:28.779 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.796+0000 7f4455ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:28.779 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.796+0000 7f4455ffb640 1 -- 172.21.15.104:0/3667791890 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f4440001690 con 0x7f44580a9f00 2024-03-29T11:24:28.779 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.796+0000 7f4455ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:28.779 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.796+0000 7f4455ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:28.779 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.796+0000 7f4455ffb640 10 monclient: dump: 2024-03-29T11:24:28.779 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:24:28.779 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:24:28.779 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:28.779 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:28.779 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:28.779 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:28.779 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:24:28.779 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:24:28.779 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:24:28.779 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:28.780 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.797+0000 7f4457fff640 5 monclient: authenticate success, global_id 4419 2024-03-29T11:24:28.780 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.797+0000 7f4457fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:28.782 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.799+0000 7f4455ffb640 1 -- 172.21.15.104:0/3667791890 <== mon.0 v1:172.21.15.104:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4440001860 con 0x7f44580a9f00 2024-03-29T11:24:28.782 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.799+0000 7f4455ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:28.782 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.799+0000 7f4457fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:28.782 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.799+0000 7f4457fff640 4 set_mon_vals no callback set 2024-03-29T11:24:28.782 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.799+0000 7f4455ffb640 1 -- 172.21.15.104:0/3667791890 <== mon.0 v1:172.21.15.104:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f4440001cb0 con 0x7f44580a9f00 2024-03-29T11:24:28.782 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.799+0000 7f4455ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:28.782 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.799+0000 7f4455ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:28.782 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.799+0000 7f4455ffb640 10 monclient: dump: 2024-03-29T11:24:28.782 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:24:28.782 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:24:28.783 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:28.783 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:28.783 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:28.783 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:28.783 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:24:28.783 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:24:28.783 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:24:28.783 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:28.783 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.800+0000 7f4457fff640 10 monclient: shutdown 2024-03-29T11:24:28.783 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.800+0000 7f4457fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:28.783 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.800+0000 7f4457fff640 1 -- 172.21.15.104:0/3667791890 >> v1:172.21.15.104:6789/0 conn(0x7f44580a9f00 legacy=0x7f445813f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:28.783 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.800+0000 7f4457fff640 1 -- 172.21.15.104:0/3667791890 shutdown_connections 2024-03-29T11:24:28.783 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.800+0000 7f4457fff640 1 -- 172.21.15.104:0/3667791890 >> 172.21.15.104:0/3667791890 conn(0x7f4458067090 msgr2=0x7f44580b4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:28.783 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.800+0000 7f4457fff640 1 -- 172.21.15.104:0/3667791890 shutdown_connections 2024-03-29T11:24:28.784 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.801+0000 7f4457fff640 1 -- 172.21.15.104:0/3667791890 wait complete. 2024-03-29T11:24:28.784 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.801+0000 7f4457fff640 10 monclient: build_initial_monmap 2024-03-29T11:24:28.784 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.801+0000 7f4457fff640 10 monclient: monmap: 2024-03-29T11:24:28.785 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:24:28.785 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:28.785 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:24:28.785 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:24:28.785 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:28.785 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:28.785 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:24:28.785 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:24:28.785 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:24:28.785 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:28.785 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.802+0000 7f4457fff640 1 librados: starting msgr at 2024-03-29T11:24:28.785 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.802+0000 7f4457fff640 1 librados: starting objecter 2024-03-29T11:24:28.785 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.802+0000 7f4457fff640 1 Processor -- start 2024-03-29T11:24:28.787 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.802+0000 7f4457fff640 1 -- start start 2024-03-29T11:24:28.787 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.802+0000 7f4457fff640 1 librados: setting wanted keys 2024-03-29T11:24:28.787 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.802+0000 7f4457fff640 1 librados: calling monclient init 2024-03-29T11:24:28.787 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.802+0000 7f4457fff640 10 monclient: init 2024-03-29T11:24:28.787 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.803+0000 7f4457fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:28.787 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.803+0000 7f4457fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:24:28.787 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.803+0000 7f4457fff640 10 monclient(hunting): picked mon.noname-b con 0x7f44580a9f00 addr v1:172.21.15.119:6789/0 2024-03-29T11:24:28.788 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.803+0000 7f4457fff640 10 monclient(hunting): picked mon.noname-c con 0x7f44580aec00 addr v1:172.21.15.104:6790/0 2024-03-29T11:24:28.788 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.803+0000 7f4457fff640 10 monclient(hunting): picked mon.noname-a con 0x7f44580b52b0 addr v1:172.21.15.104:6789/0 2024-03-29T11:24:28.788 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.803+0000 7f4457fff640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44581432d0 con 0x7f44580b52b0 2024-03-29T11:24:28.788 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.803+0000 7f4457fff640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f445814a220 con 0x7f44580a9f00 2024-03-29T11:24:28.788 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.803+0000 7f4457fff640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44581e7b60 con 0x7f44580aec00 2024-03-29T11:24:28.788 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.803+0000 7f4457fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:28.788 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.803+0000 7f4457fff640 10 monclient(hunting): authenticate will time out at 1250.054747s 2024-03-29T11:24:28.790 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.808+0000 7f44567fc640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f44580aec00 0x7f4458140fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:52339/0 (socket says 172.21.15.104:52339) 2024-03-29T11:24:28.790 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.808+0000 7f44567fc640 1 -- 172.21.15.104:0/1951393041 learned_addr learned my addr 172.21.15.104:0/1951393041 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:24:28.790 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.808+0000 7f444f7fe640 10 client.?.objecter ms_handle_connect 0x7f44580aec00 2024-03-29T11:24:28.790 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.808+0000 7f444f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:28.794 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.811+0000 7f444f7fe640 1 -- 172.21.15.104:0/1951393041 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2728299292 0 0) 0x7f44581e7b60 con 0x7f44580aec00 2024-03-29T11:24:28.794 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.811+0000 7f444f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:28.794 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.811+0000 7f444f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:28.794 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.811+0000 7f444f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:28.794 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.811+0000 7f444f7fe640 10 monclient(hunting): my global_id is 4424 2024-03-29T11:24:28.794 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.811+0000 7f444f7fe640 1 -- 172.21.15.104:0/1951393041 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4438004eb0 con 0x7f44580aec00 2024-03-29T11:24:28.794 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.811+0000 7f444f7fe640 10 client.?.objecter ms_handle_connect 0x7f44580a9f00 2024-03-29T11:24:28.794 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.811+0000 7f444f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:28.794 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.811+0000 7f444f7fe640 10 client.?.objecter ms_handle_connect 0x7f44580b52b0 2024-03-29T11:24:28.794 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.811+0000 7f444f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:28.795 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.812+0000 7f444f7fe640 1 -- 172.21.15.104:0/1951393041 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2905561183 0 0) 0x7f4438004eb0 con 0x7f44580aec00 2024-03-29T11:24:28.795 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.812+0000 7f444f7fe640 1 -- 172.21.15.104:0/1951393041 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4438005300 con 0x7f44580aec00 2024-03-29T11:24:28.795 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.812+0000 7f444f7fe640 1 -- 172.21.15.104:0/1951393041 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1233763461 0 0) 0x7f44581432d0 con 0x7f44580b52b0 2024-03-29T11:24:28.795 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.812+0000 7f444f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:28.795 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.812+0000 7f444f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:28.795 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.812+0000 7f444f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:28.795 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.812+0000 7f444f7fe640 10 monclient(hunting): my global_id is 4422 2024-03-29T11:24:28.795 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.812+0000 7f444f7fe640 1 -- 172.21.15.104:0/1951393041 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4438004eb0 con 0x7f44580b52b0 2024-03-29T11:24:28.795 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.812+0000 7f444f7fe640 1 -- 172.21.15.104:0/1951393041 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f4448001cb0 con 0x7f44580aec00 2024-03-29T11:24:28.795 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.812+0000 7f444f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:28.795 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.812+0000 7f444f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:28.795 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.812+0000 7f444f7fe640 10 monclient(hunting): dump: 2024-03-29T11:24:28.795 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:24:28.795 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:24:28.796 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:28.796 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:28.796 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:28.796 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:28.796 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:24:28.796 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:24:28.796 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:24:28.796 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:28.796 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.812+0000 7f444f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:28.796 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.812+0000 7f444f7fe640 1 -- 172.21.15.104:0/1951393041 <== mon.2 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1994985118 0 0) 0x7f4438005300 con 0x7f44580aec00 2024-03-29T11:24:28.796 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.812+0000 7f444f7fe640 1 -- 172.21.15.104:0/1951393041 >> v1:172.21.15.119:6789/0 conn(0x7f44580a9f00 legacy=0x7f44581e7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:28.796 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.812+0000 7f444f7fe640 1 -- 172.21.15.104:0/1951393041 >> v1:172.21.15.104:6789/0 conn(0x7f44580b52b0 legacy=0x7f44581416f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:28.796 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.812+0000 7f444f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:24:28.796 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.812+0000 7f444f7fe640 1 monclient: found mon.c 2024-03-29T11:24:28.797 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.812+0000 7f444f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:28.797 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.812+0000 7f444f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:24:28.797 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.812+0000 7f444f7fe640 1 -- 172.21.15.104:0/1951393041 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44581e7cf0 con 0x7f44580aec00 2024-03-29T11:24:28.797 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.812+0000 7f444f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:24:28.797 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.812+0000 7f444f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:24:28.797 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.812+0000 7f444f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:28.798 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.813+0000 7f444f7fe640 1 -- 172.21.15.104:0/1951393041 <== mon.2 v1:172.21.15.104:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4448003970 con 0x7f44580aec00 2024-03-29T11:24:28.798 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.813+0000 7f444f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:28.798 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.813+0000 7f444f7fe640 1 -- 172.21.15.104:0/1951393041 <== mon.2 v1:172.21.15.104:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f4448003dc0 con 0x7f44580aec00 2024-03-29T11:24:28.798 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.813+0000 7f444f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:28.798 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.813+0000 7f444f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:28.798 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.813+0000 7f444f7fe640 10 monclient: dump: 2024-03-29T11:24:28.798 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:24:28.798 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:24:28.798 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:28.798 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:28.799 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:28.799 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:28.799 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:24:28.799 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:24:28.799 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:24:28.799 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:28.799 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.813+0000 7f4454ff9640 4 set_mon_vals no callback set 2024-03-29T11:24:28.799 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.814+0000 7f4457fff640 5 monclient: authenticate success, global_id 4424 2024-03-29T11:24:28.799 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.814+0000 7f4457fff640 10 monclient: _renew_subs 2024-03-29T11:24:28.799 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.814+0000 7f4457fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:24:28.799 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.814+0000 7f4457fff640 1 -- 172.21.15.104:0/1951393041 --> v1:172.21.15.104:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44581ef730 con 0x7f44580aec00 2024-03-29T11:24:28.799 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.814+0000 7f4457fff640 10 client.4424.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:28.799 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.814+0000 7f4457fff640 10 monclient: _renew_subs 2024-03-29T11:24:28.800 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.814+0000 7f4457fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:24:28.800 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.814+0000 7f4457fff640 1 -- 172.21.15.104:0/1951393041 --> v1:172.21.15.104:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f44581efaa0 con 0x7f44580aec00 2024-03-29T11:24:28.800 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.814+0000 7f4457fff640 1 librados: init done 2024-03-29T11:24:28.800 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.815+0000 7f4457fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:28.800 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.815+0000 7f4457fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:28.800 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.815+0000 7f4457fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:24:28.800 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.815+0000 7f4457fff640 1 -- 172.21.15.104:0/1951393041 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f445804f730 con 0x7f44580aec00 2024-03-29T11:24:28.801 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.817+0000 7f444f7fe640 1 -- 172.21.15.104:0/1951393041 <== mon.2 v1:172.21.15.104:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f4448003fa0 con 0x7f44580aec00 2024-03-29T11:24:28.801 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.818+0000 7f444f7fe640 1 -- 172.21.15.104:0/1951393041 <== mon.2 v1:172.21.15.104:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f4448003be0 con 0x7f44580aec00 2024-03-29T11:24:28.801 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.818+0000 7f444f7fe640 10 client.4424.objecter ms_dispatch 0x7f4458000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:28.801 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.818+0000 7f444f7fe640 3 client.4424.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:28.801 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.818+0000 7f444f7fe640 3 client.4424.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:28.801 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.818+0000 7f444f7fe640 20 client.4424.objecter dump_active .. 0 homeless 2024-03-29T11:24:28.801 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.819+0000 7f4456ffd640 10 monclient: get_auth_request con 0x7f443806e050 auth_method 0 2024-03-29T11:24:28.804 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.823+0000 7f444f7fe640 10 client.4424.objecter ms_handle_connect 0x7f443806e050 2024-03-29T11:24:28.809 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.828+0000 7f444f7fe640 1 -- 172.21.15.104:0/1951393041 <== mon.2 v1:172.21.15.104:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4448050bf0 con 0x7f44580aec00 2024-03-29T11:24:28.809 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.828+0000 7f444f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:28.809 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:28.828+0000 7f444f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:29.142 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:29.158+0000 7f4457fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:29.142 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:29.158+0000 7f4457fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:29.142 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:29.158+0000 7f4457fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:24:29.142 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:29.158+0000 7f4457fff640 1 -- 172.21.15.104:0/1951393041 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f44580b3c30 con 0x7f44580aec00 2024-03-29T11:24:29.143 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:29.161+0000 7f444f7fe640 1 -- 172.21.15.104:0/1951393041 <== mon.2 v1:172.21.15.104:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f4448050bf0 con 0x7f44580aec00 2024-03-29T11:24:29.143 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:29.161+0000 7f444f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:29.143 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:29.161+0000 7f444f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:29.143 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:24:29.144 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:29.147 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:29.165+0000 7f4457fff640 10 librados: watch_flush enter 2024-03-29T11:24:29.147 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:29.165+0000 7f4457fff640 10 librados: watch_flush exit 2024-03-29T11:24:29.147 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:29.165+0000 7f4457fff640 20 client.4424.objecter shutdown clearing up homeless session... 2024-03-29T11:24:29.147 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:29.165+0000 7f4457fff640 10 client.4424.objecter successfully canceled tick 2024-03-29T11:24:29.150 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:29.165+0000 7f4457fff640 1 -- 172.21.15.104:0/1951393041 >> v1:172.21.15.104:6816/3341126660 conn(0x7f443806e050 legacy=0x7f4438070510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:29.150 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:29.165+0000 7f4457fff640 10 monclient: shutdown 2024-03-29T11:24:29.150 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:29.165+0000 7f4457fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:29.150 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:29.165+0000 7f4457fff640 1 -- 172.21.15.104:0/1951393041 >> v1:172.21.15.104:6790/0 conn(0x7f44580aec00 legacy=0x7f4458140fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:29.151 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:29.167+0000 7f4457fff640 1 -- 172.21.15.104:0/1951393041 shutdown_connections 2024-03-29T11:24:29.151 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:29.167+0000 7f4457fff640 1 -- 172.21.15.104:0/1951393041 >> 172.21.15.104:0/1951393041 conn(0x7f4458067090 msgr2=0x7f445806ab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:29.151 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:29.167+0000 7f4457fff640 1 -- 172.21.15.104:0/1951393041 shutdown_connections 2024-03-29T11:24:29.151 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:29.167+0000 7f4457fff640 1 -- 172.21.15.104:0/1951393041 wait complete. 2024-03-29T11:24:29.151 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:29.167+0000 7f4457fff640 1 librados: shutdown 2024-03-29T11:24:29.172 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:35.979 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:36.124 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.142+0000 7f2e33dc8640 10 monclient: get_monmap_and_config 2024-03-29T11:24:36.124 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.142+0000 7f2e33dc8640 10 monclient: build_initial_monmap 2024-03-29T11:24:36.125 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.142+0000 7f2e33dc8640 10 monclient: monmap: 2024-03-29T11:24:36.125 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:24:36.125 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:24:36.125 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:24:36.143441+0000 2024-03-29T11:24:36.125 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:24:36.143441+0000 2024-03-29T11:24:36.125 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:36.125 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:36.125 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:24:36.125 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:24:36.125 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:24:36.125 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:36.128 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.145+0000 7f2e33dc8640 1 Processor -- start 2024-03-29T11:24:36.128 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.145+0000 7f2e33dc8640 1 -- start start 2024-03-29T11:24:36.128 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.145+0000 7f2e33dc8640 10 monclient: init 2024-03-29T11:24:36.128 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.146+0000 7f2e33dc8640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:36.128 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.146+0000 7f2e33dc8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:36.134 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.147+0000 7f2e33dc8640 10 monclient(hunting): picked mon.noname-a con 0x7f2e2c0b0860 addr v1:172.21.15.104:6789/0 2024-03-29T11:24:36.134 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.147+0000 7f2e33dc8640 10 monclient(hunting): picked mon.noname-c con 0x7f2e2c13ef30 addr v1:172.21.15.119:6789/0 2024-03-29T11:24:36.134 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.147+0000 7f2e33dc8640 10 monclient(hunting): picked mon.noname-b con 0x7f2e2c06d9d0 addr v1:172.21.15.104:6790/0 2024-03-29T11:24:36.134 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.147+0000 7f2e33dc8640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e2c0b0c40 con 0x7f2e2c0b0860 2024-03-29T11:24:36.134 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.147+0000 7f2e33dc8640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e2c0aed10 con 0x7f2e2c13ef30 2024-03-29T11:24:36.134 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.147+0000 7f2e33dc8640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e2c0ab1b0 con 0x7f2e2c06d9d0 2024-03-29T11:24:36.135 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.147+0000 7f2e33dc8640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:36.135 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.147+0000 7f2e33dc8640 10 monclient(hunting): authenticate will time out at 1257.398623s 2024-03-29T11:24:36.135 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.152+0000 7f2e32365640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f2e2c06d9d0 0x7f2e2c14a300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:55705/0 (socket says 172.21.15.104:55705) 2024-03-29T11:24:36.135 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.152+0000 7f2e32365640 1 -- 172.21.15.104:0/418339103 learned_addr learned my addr 172.21.15.104:0/418339103 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:24:36.135 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.152+0000 7f2e30b62640 1 -- 172.21.15.104:0/418339103 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2687217279 0 0) 0x7f2e2c0ab1b0 con 0x7f2e2c06d9d0 2024-03-29T11:24:36.135 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.152+0000 7f2e30b62640 10 monclient(hunting): _negotiate 2024-03-29T11:24:36.135 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.152+0000 7f2e30b62640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:36.135 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.152+0000 7f2e30b62640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:36.135 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.152+0000 7f2e30b62640 10 monclient(hunting): my global_id is 4427 2024-03-29T11:24:36.135 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.152+0000 7f2e30b62640 1 -- 172.21.15.104:0/418339103 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e18003520 con 0x7f2e2c06d9d0 2024-03-29T11:24:36.135 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.152+0000 7f2e30b62640 1 -- 172.21.15.104:0/418339103 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3596697034 0 0) 0x7f2e18003520 con 0x7f2e2c06d9d0 2024-03-29T11:24:36.135 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.152+0000 7f2e30b62640 1 -- 172.21.15.104:0/418339103 >> v1:172.21.15.119:6789/0 conn(0x7f2e2c13ef30 legacy=0x7f2e2c0af1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:24:36.136 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.154+0000 7f2e30b62640 1 -- 172.21.15.104:0/418339103 >> v1:172.21.15.104:6789/0 conn(0x7f2e2c0b0860 legacy=0x7f2e2c13e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:36.136 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.154+0000 7f2e30b62640 10 monclient: _finish_hunting 0 2024-03-29T11:24:36.137 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.154+0000 7f2e30b62640 1 monclient: found mon.noname-b 2024-03-29T11:24:36.137 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.154+0000 7f2e30b62640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:36.137 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.154+0000 7f2e30b62640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.104:6790/0 2024-03-29T11:24:36.137 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.154+0000 7f2e30b62640 1 -- 172.21.15.104:0/418339103 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e2c0ab490 con 0x7f2e2c06d9d0 2024-03-29T11:24:36.137 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.154+0000 7f2e30b62640 10 monclient: _finish_auth 0 2024-03-29T11:24:36.137 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.154+0000 7f2e30b62640 10 monclient: _check_auth_tickets 2024-03-29T11:24:36.137 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.154+0000 7f2e30b62640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:36.137 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.154+0000 7f2e30b62640 1 -- 172.21.15.104:0/418339103 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f2e1c001690 con 0x7f2e2c06d9d0 2024-03-29T11:24:36.137 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.154+0000 7f2e30b62640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:36.137 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.154+0000 7f2e30b62640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:36.137 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.154+0000 7f2e30b62640 10 monclient: dump: 2024-03-29T11:24:36.137 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:24:36.137 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:24:36.138 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:36.138 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:36.138 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:36.138 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:36.138 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:24:36.138 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:24:36.138 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:24:36.138 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:36.138 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.154+0000 7f2e33dc8640 5 monclient: authenticate success, global_id 4427 2024-03-29T11:24:36.138 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.154+0000 7f2e33dc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:36.139 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.154+0000 7f2e30b62640 1 -- 172.21.15.104:0/418339103 <== mon.2 v1:172.21.15.104:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e1c001860 con 0x7f2e2c06d9d0 2024-03-29T11:24:36.139 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.154+0000 7f2e30b62640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:36.139 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.154+0000 7f2e30b62640 1 -- 172.21.15.104:0/418339103 <== mon.2 v1:172.21.15.104:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f2e1c001cb0 con 0x7f2e2c06d9d0 2024-03-29T11:24:36.139 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.154+0000 7f2e30b62640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:36.139 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.154+0000 7f2e30b62640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:36.139 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.154+0000 7f2e30b62640 10 monclient: dump: 2024-03-29T11:24:36.139 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:24:36.139 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:24:36.139 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:36.139 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:36.139 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:36.140 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:36.140 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:24:36.140 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:24:36.140 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:24:36.140 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:36.140 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.154+0000 7f2e33dc8640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:36.140 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.154+0000 7f2e33dc8640 4 set_mon_vals no callback set 2024-03-29T11:24:36.142 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.155+0000 7f2e33dc8640 10 monclient: shutdown 2024-03-29T11:24:36.142 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.155+0000 7f2e33dc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:36.142 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.155+0000 7f2e33dc8640 1 -- 172.21.15.104:0/418339103 >> v1:172.21.15.104:6790/0 conn(0x7f2e2c06d9d0 legacy=0x7f2e2c14a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:36.142 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.156+0000 7f2e33dc8640 1 -- 172.21.15.104:0/418339103 shutdown_connections 2024-03-29T11:24:36.142 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.156+0000 7f2e33dc8640 1 -- 172.21.15.104:0/418339103 >> 172.21.15.104:0/418339103 conn(0x7f2e2c06b660 msgr2=0x7f2e2c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:36.143 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.157+0000 7f2e33dc8640 1 -- 172.21.15.104:0/418339103 shutdown_connections 2024-03-29T11:24:36.143 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.157+0000 7f2e33dc8640 1 -- 172.21.15.104:0/418339103 wait complete. 2024-03-29T11:24:36.143 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.157+0000 7f2e33dc8640 10 monclient: build_initial_monmap 2024-03-29T11:24:36.143 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.157+0000 7f2e33dc8640 10 monclient: monmap: 2024-03-29T11:24:36.143 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:24:36.143 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:36.143 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:24:36.143 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:24:36.143 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:36.144 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:36.144 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:24:36.144 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:24:36.144 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:24:36.144 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:36.144 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.158+0000 7f2e33dc8640 1 librados: starting msgr at 2024-03-29T11:24:36.144 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.158+0000 7f2e33dc8640 1 librados: starting objecter 2024-03-29T11:24:36.144 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.158+0000 7f2e33dc8640 1 Processor -- start 2024-03-29T11:24:36.144 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.158+0000 7f2e33dc8640 1 -- start start 2024-03-29T11:24:36.144 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.158+0000 7f2e33dc8640 1 librados: setting wanted keys 2024-03-29T11:24:36.144 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.158+0000 7f2e33dc8640 1 librados: calling monclient init 2024-03-29T11:24:36.145 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.158+0000 7f2e33dc8640 10 monclient: init 2024-03-29T11:24:36.145 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.159+0000 7f2e33dc8640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:36.145 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.159+0000 7f2e33dc8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:24:36.145 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.159+0000 7f2e33dc8640 10 monclient(hunting): picked mon.noname-c con 0x7f2e2c06d9d0 addr v1:172.21.15.104:6790/0 2024-03-29T11:24:36.145 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.159+0000 7f2e33dc8640 10 monclient(hunting): picked mon.noname-a con 0x7f2e2c0b0860 addr v1:172.21.15.104:6789/0 2024-03-29T11:24:36.145 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.159+0000 7f2e33dc8640 10 monclient(hunting): picked mon.noname-b con 0x7f2e2c13ef30 addr v1:172.21.15.119:6789/0 2024-03-29T11:24:36.145 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.159+0000 7f2e33dc8640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e2c0b3d70 con 0x7f2e2c0b0860 2024-03-29T11:24:36.145 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.159+0000 7f2e33dc8640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e2c0b3390 con 0x7f2e2c13ef30 2024-03-29T11:24:36.145 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.159+0000 7f2e33dc8640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e2c0b2880 con 0x7f2e2c06d9d0 2024-03-29T11:24:36.145 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.159+0000 7f2e33dc8640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:36.145 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.159+0000 7f2e33dc8640 10 monclient(hunting): authenticate will time out at 1257.410365s 2024-03-29T11:24:36.146 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.164+0000 7f2e31363640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f2e2c0b0860 0x7f2e2c13a640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:45957/0 (socket says 172.21.15.104:45957) 2024-03-29T11:24:36.146 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.164+0000 7f2e31363640 1 -- 172.21.15.104:0/853490849 learned_addr learned my addr 172.21.15.104:0/853490849 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:24:36.146 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.164+0000 7f2e227fc640 10 client.?.objecter ms_handle_connect 0x7f2e2c0b0860 2024-03-29T11:24:36.146 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.164+0000 7f2e227fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:36.147 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.165+0000 7f2e227fc640 1 -- 172.21.15.104:0/853490849 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 285985181 0 0) 0x7f2e2c0b3d70 con 0x7f2e2c0b0860 2024-03-29T11:24:36.147 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.165+0000 7f2e227fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:36.147 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.165+0000 7f2e227fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:36.147 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.165+0000 7f2e227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:36.147 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.165+0000 7f2e227fc640 10 monclient(hunting): my global_id is 4425 2024-03-29T11:24:36.147 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.165+0000 7f2e227fc640 1 -- 172.21.15.104:0/853490849 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e10004eb0 con 0x7f2e2c0b0860 2024-03-29T11:24:36.148 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.165+0000 7f2e227fc640 1 -- 172.21.15.104:0/853490849 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2736682607 0 0) 0x7f2e10004eb0 con 0x7f2e2c0b0860 2024-03-29T11:24:36.148 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.165+0000 7f2e227fc640 1 -- 172.21.15.104:0/853490849 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e2c0b3d70 con 0x7f2e2c0b0860 2024-03-29T11:24:36.148 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.165+0000 7f2e227fc640 1 -- 172.21.15.104:0/853490849 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f2e280017d0 con 0x7f2e2c0b0860 2024-03-29T11:24:36.148 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.165+0000 7f2e227fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:36.148 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.165+0000 7f2e227fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:36.148 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.165+0000 7f2e227fc640 10 monclient(hunting): dump: 2024-03-29T11:24:36.148 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:24:36.148 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:24:36.148 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:36.148 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:36.148 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:36.148 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:36.148 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:24:36.148 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:24:36.148 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:24:36.149 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:36.149 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.165+0000 7f2e227fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:36.149 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.165+0000 7f2e227fc640 1 -- 172.21.15.104:0/853490849 <== mon.0 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3500790403 0 0) 0x7f2e2c0b3d70 con 0x7f2e2c0b0860 2024-03-29T11:24:36.149 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.166+0000 7f2e227fc640 1 -- 172.21.15.104:0/853490849 >> v1:172.21.15.104:6790/0 conn(0x7f2e2c06d9d0 legacy=0x7f2e2c0b1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:36.149 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.166+0000 7f2e227fc640 1 -- 172.21.15.104:0/853490849 >> v1:172.21.15.119:6789/0 conn(0x7f2e2c13ef30 legacy=0x7f2e2c1ded70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:36.149 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.166+0000 7f2e227fc640 10 monclient: _finish_hunting 0 2024-03-29T11:24:36.149 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.166+0000 7f2e227fc640 1 monclient: found mon.a 2024-03-29T11:24:36.149 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.166+0000 7f2e227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:36.150 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.166+0000 7f2e227fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:24:36.150 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.166+0000 7f2e227fc640 1 -- 172.21.15.104:0/853490849 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e2c0b3600 con 0x7f2e2c0b0860 2024-03-29T11:24:36.150 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.166+0000 7f2e227fc640 10 monclient: _finish_auth 0 2024-03-29T11:24:36.150 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.166+0000 7f2e227fc640 10 monclient: _check_auth_tickets 2024-03-29T11:24:36.150 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.166+0000 7f2e227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:36.150 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.166+0000 7f2e33dc8640 5 monclient: authenticate success, global_id 4425 2024-03-29T11:24:36.150 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.166+0000 7f2e33dc8640 10 monclient: _renew_subs 2024-03-29T11:24:36.150 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.166+0000 7f2e33dc8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:24:36.150 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.166+0000 7f2e33dc8640 1 -- 172.21.15.104:0/853490849 --> v1:172.21.15.104:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e2c0b25a0 con 0x7f2e2c0b0860 2024-03-29T11:24:36.150 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.166+0000 7f2e33dc8640 10 client.4425.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:36.150 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.166+0000 7f2e33dc8640 10 monclient: _renew_subs 2024-03-29T11:24:36.150 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.166+0000 7f2e33dc8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:24:36.150 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.166+0000 7f2e33dc8640 1 -- 172.21.15.104:0/853490849 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e2c0b41e0 con 0x7f2e2c0b0860 2024-03-29T11:24:36.150 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.166+0000 7f2e33dc8640 1 librados: init done 2024-03-29T11:24:36.151 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.168+0000 7f2e227fc640 1 -- 172.21.15.104:0/853490849 <== mon.0 v1:172.21.15.104:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e28001930 con 0x7f2e2c0b0860 2024-03-29T11:24:36.151 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.168+0000 7f2e227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:36.151 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.168+0000 7f2e227fc640 1 -- 172.21.15.104:0/853490849 <== mon.0 v1:172.21.15.104:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f2e28004a40 con 0x7f2e2c0b0860 2024-03-29T11:24:36.151 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.168+0000 7f2e227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:36.151 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.168+0000 7f2e227fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:36.151 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.168+0000 7f2e227fc640 10 monclient: dump: 2024-03-29T11:24:36.152 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:24:36.152 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:24:36.152 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:36.152 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:36.152 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:36.152 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:36.152 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:24:36.152 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:24:36.152 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:24:36.152 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:36.152 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.168+0000 7f2e17fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:36.153 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.168+0000 7f2e17fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:36.153 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.168+0000 7f2e17fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:24:36.153 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.168+0000 7f2e17fff640 1 -- 172.21.15.104:0/853490849 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e2c075f10 con 0x7f2e2c0b0860 2024-03-29T11:24:36.153 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.168+0000 7f2e23fff640 4 set_mon_vals no callback set 2024-03-29T11:24:36.153 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.171+0000 7f2e227fc640 1 -- 172.21.15.104:0/853490849 <== mon.0 v1:172.21.15.104:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f2e2801aad0 con 0x7f2e2c0b0860 2024-03-29T11:24:36.153 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.172+0000 7f2e31b64640 10 monclient: get_auth_request con 0x7f2e1006dc40 auth_method 0 2024-03-29T11:24:36.154 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.172+0000 7f2e227fc640 1 -- 172.21.15.104:0/853490849 <== mon.0 v1:172.21.15.104:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f2e28054910 con 0x7f2e2c0b0860 2024-03-29T11:24:36.154 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.172+0000 7f2e227fc640 10 client.4425.objecter ms_dispatch 0x7f2e2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:36.155 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.172+0000 7f2e227fc640 3 client.4425.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:36.155 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.172+0000 7f2e227fc640 3 client.4425.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:36.155 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.172+0000 7f2e227fc640 20 client.4425.objecter dump_active .. 0 homeless 2024-03-29T11:24:36.156 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.172+0000 7f2e227fc640 10 client.4425.objecter ms_handle_connect 0x7f2e1006dc40 2024-03-29T11:24:36.166 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.181+0000 7f2e227fc640 1 -- 172.21.15.104:0/853490849 <== mon.0 v1:172.21.15.104:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2e28050250 con 0x7f2e2c0b0860 2024-03-29T11:24:36.166 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.181+0000 7f2e227fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:36.167 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.181+0000 7f2e227fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:36.491 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.509+0000 7f2e17fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:36.491 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.509+0000 7f2e17fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:36.491 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.509+0000 7f2e17fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:24:36.491 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.509+0000 7f2e17fff640 1 -- 172.21.15.104:0/853490849 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2e2c072330 con 0x7f2e2c0b0860 2024-03-29T11:24:36.493 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.511+0000 7f2e227fc640 1 -- 172.21.15.104:0/853490849 <== mon.0 v1:172.21.15.104:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f2e28052350 con 0x7f2e2c0b0860 2024-03-29T11:24:36.493 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.511+0000 7f2e227fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:36.493 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.511+0000 7f2e227fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:36.494 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:24:36.494 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:36.496 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.514+0000 7f2e17fff640 10 librados: watch_flush enter 2024-03-29T11:24:36.500 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.517+0000 7f2e17fff640 10 librados: watch_flush exit 2024-03-29T11:24:36.500 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.517+0000 7f2e17fff640 20 client.4425.objecter shutdown clearing up homeless session... 2024-03-29T11:24:36.500 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.517+0000 7f2e17fff640 10 client.4425.objecter successfully canceled tick 2024-03-29T11:24:36.500 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.518+0000 7f2e17fff640 1 -- 172.21.15.104:0/853490849 >> v1:172.21.15.104:6816/3341126660 conn(0x7f2e1006dc40 legacy=0x7f2e10070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:36.500 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.518+0000 7f2e17fff640 10 monclient: shutdown 2024-03-29T11:24:36.500 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.518+0000 7f2e17fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:36.500 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.518+0000 7f2e17fff640 1 -- 172.21.15.104:0/853490849 >> v1:172.21.15.104:6789/0 conn(0x7f2e2c0b0860 legacy=0x7f2e2c13a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:36.501 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.518+0000 7f2e17fff640 1 -- 172.21.15.104:0/853490849 shutdown_connections 2024-03-29T11:24:36.501 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.518+0000 7f2e17fff640 1 -- 172.21.15.104:0/853490849 >> 172.21.15.104:0/853490849 conn(0x7f2e2c06b660 msgr2=0x7f2e2c06ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:36.501 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.518+0000 7f2e17fff640 1 -- 172.21.15.104:0/853490849 shutdown_connections 2024-03-29T11:24:36.501 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.518+0000 7f2e17fff640 1 -- 172.21.15.104:0/853490849 wait complete. 2024-03-29T11:24:36.501 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:36.518+0000 7f2e17fff640 1 librados: shutdown 2024-03-29T11:24:36.516 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:43.323 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:43.477 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.494+0000 7f7f59e82640 10 monclient: get_monmap_and_config 2024-03-29T11:24:43.477 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.494+0000 7f7f59e82640 10 monclient: build_initial_monmap 2024-03-29T11:24:43.477 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.494+0000 7f7f59e82640 10 monclient: monmap: 2024-03-29T11:24:43.477 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:24:43.477 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:24:43.477 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:24:43.495610+0000 2024-03-29T11:24:43.477 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:24:43.495610+0000 2024-03-29T11:24:43.477 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:43.477 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:43.477 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:24:43.477 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:24:43.478 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:24:43.478 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:43.478 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.495+0000 7f7f59e82640 1 Processor -- start 2024-03-29T11:24:43.478 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.495+0000 7f7f59e82640 1 -- start start 2024-03-29T11:24:43.478 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.495+0000 7f7f59e82640 10 monclient: init 2024-03-29T11:24:43.479 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.496+0000 7f7f59e82640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:43.479 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.496+0000 7f7f59e82640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:24:43.479 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.496+0000 7f7f59e82640 10 monclient(hunting): picked mon.noname-a con 0x7f7f540b0770 addr v1:172.21.15.104:6789/0 2024-03-29T11:24:43.479 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.496+0000 7f7f59e82640 10 monclient(hunting): picked mon.noname-b con 0x7f7f5414a700 addr v1:172.21.15.104:6790/0 2024-03-29T11:24:43.479 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.496+0000 7f7f59e82640 10 monclient(hunting): picked mon.noname-c con 0x7f7f540ac5a0 addr v1:172.21.15.119:6789/0 2024-03-29T11:24:43.479 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.496+0000 7f7f59e82640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f540aba70 con 0x7f7f540b0770 2024-03-29T11:24:43.479 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.496+0000 7f7f59e82640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f540aecb0 con 0x7f7f540ac5a0 2024-03-29T11:24:43.479 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.496+0000 7f7f59e82640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f540af150 con 0x7f7f5414a700 2024-03-29T11:24:43.479 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.496+0000 7f7f59e82640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:43.479 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.496+0000 7f7f59e82640 10 monclient(hunting): authenticate will time out at 1264.747878s 2024-03-29T11:24:43.482 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.497+0000 7f7f53fff640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f7f540ac5a0 0x7f7f54151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:50046/0 (socket says 172.21.15.104:50046) 2024-03-29T11:24:43.482 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.497+0000 7f7f53fff640 1 -- 172.21.15.104:0/1130899009 learned_addr learned my addr 172.21.15.104:0/1130899009 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:24:43.482 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.497+0000 7f7f527fc640 1 -- 172.21.15.104:0/1130899009 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3020280995 0 0) 0x7f7f540aecb0 con 0x7f7f540ac5a0 2024-03-29T11:24:43.482 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.497+0000 7f7f527fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:43.482 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.497+0000 7f7f527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:43.482 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.497+0000 7f7f527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:43.482 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.497+0000 7f7f527fc640 10 monclient(hunting): my global_id is 4402 2024-03-29T11:24:43.482 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.497+0000 7f7f527fc640 1 -- 172.21.15.104:0/1130899009 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f3c003520 con 0x7f7f540ac5a0 2024-03-29T11:24:43.482 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.497+0000 7f7f527fc640 1 -- 172.21.15.104:0/1130899009 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3657327198 0 0) 0x7f7f540aba70 con 0x7f7f540b0770 2024-03-29T11:24:43.483 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.497+0000 7f7f527fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:43.483 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.497+0000 7f7f527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:43.483 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.497+0000 7f7f527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:43.483 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.497+0000 7f7f527fc640 10 monclient(hunting): my global_id is 4428 2024-03-29T11:24:43.483 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.497+0000 7f7f527fc640 1 -- 172.21.15.104:0/1130899009 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f540aecb0 con 0x7f7f540b0770 2024-03-29T11:24:43.483 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.498+0000 7f7f527fc640 1 -- 172.21.15.104:0/1130899009 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3360433087 0 0) 0x7f7f3c003520 con 0x7f7f540ac5a0 2024-03-29T11:24:43.483 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.498+0000 7f7f527fc640 1 -- 172.21.15.104:0/1130899009 >> v1:172.21.15.104:6790/0 conn(0x7f7f5414a700 legacy=0x7f7f540abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:43.483 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.498+0000 7f7f527fc640 1 -- 172.21.15.104:0/1130899009 >> v1:172.21.15.104:6789/0 conn(0x7f7f540b0770 legacy=0x7f7f54149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:43.483 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.498+0000 7f7f527fc640 10 monclient: _finish_hunting 0 2024-03-29T11:24:43.483 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.498+0000 7f7f527fc640 1 monclient: found mon.noname-c 2024-03-29T11:24:43.483 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.498+0000 7f7f527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:43.483 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.498+0000 7f7f527fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.119:6789/0 2024-03-29T11:24:43.483 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.498+0000 7f7f527fc640 1 -- 172.21.15.104:0/1130899009 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f540ac9d0 con 0x7f7f540ac5a0 2024-03-29T11:24:43.483 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.498+0000 7f7f527fc640 10 monclient: _finish_auth 0 2024-03-29T11:24:43.483 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.498+0000 7f7f527fc640 10 monclient: _check_auth_tickets 2024-03-29T11:24:43.484 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.498+0000 7f7f527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:43.484 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.498+0000 7f7f527fc640 1 -- 172.21.15.104:0/1130899009 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f7f480016b0 con 0x7f7f540ac5a0 2024-03-29T11:24:43.484 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.498+0000 7f7f527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:43.484 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.498+0000 7f7f527fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:43.484 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.498+0000 7f7f527fc640 10 monclient: dump: 2024-03-29T11:24:43.484 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:24:43.484 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:24:43.484 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:43.484 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:43.484 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:43.484 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:43.484 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:24:43.484 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:24:43.484 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:24:43.485 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:43.485 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.498+0000 7f7f59e82640 5 monclient: authenticate success, global_id 4402 2024-03-29T11:24:43.485 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.498+0000 7f7f59e82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:43.485 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.498+0000 7f7f527fc640 1 -- 172.21.15.104:0/1130899009 <== mon.1 v1:172.21.15.119:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f48001810 con 0x7f7f540ac5a0 2024-03-29T11:24:43.485 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.498+0000 7f7f527fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:43.485 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.498+0000 7f7f527fc640 1 -- 172.21.15.104:0/1130899009 <== mon.1 v1:172.21.15.119:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f7f48001be0 con 0x7f7f540ac5a0 2024-03-29T11:24:43.485 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.498+0000 7f7f527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:43.485 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.498+0000 7f7f527fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:43.485 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.498+0000 7f7f527fc640 10 monclient: dump: 2024-03-29T11:24:43.485 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:24:43.485 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:24:43.485 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:43.485 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:43.485 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:43.485 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:43.486 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:24:43.486 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:24:43.486 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:24:43.486 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:43.486 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.498+0000 7f7f59e82640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:43.486 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.498+0000 7f7f59e82640 4 set_mon_vals no callback set 2024-03-29T11:24:43.486 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.499+0000 7f7f59e82640 10 monclient: shutdown 2024-03-29T11:24:43.486 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.499+0000 7f7f59e82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:43.486 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.499+0000 7f7f59e82640 1 -- 172.21.15.104:0/1130899009 >> v1:172.21.15.119:6789/0 conn(0x7f7f540ac5a0 legacy=0x7f7f54151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:43.486 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.499+0000 7f7f59e82640 1 -- 172.21.15.104:0/1130899009 shutdown_connections 2024-03-29T11:24:43.486 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.499+0000 7f7f59e82640 1 -- 172.21.15.104:0/1130899009 >> 172.21.15.104:0/1130899009 conn(0x7f7f5406b660 msgr2=0x7f7f5406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:43.486 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.499+0000 7f7f59e82640 1 -- 172.21.15.104:0/1130899009 shutdown_connections 2024-03-29T11:24:43.486 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.499+0000 7f7f59e82640 1 -- 172.21.15.104:0/1130899009 wait complete. 2024-03-29T11:24:43.492 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.500+0000 7f7f59e82640 10 monclient: build_initial_monmap 2024-03-29T11:24:43.492 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.500+0000 7f7f59e82640 10 monclient: monmap: 2024-03-29T11:24:43.492 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:24:43.492 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:43.492 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:24:43.492 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:24:43.493 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:43.493 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:43.493 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:24:43.493 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:24:43.493 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:24:43.493 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:43.493 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.501+0000 7f7f59e82640 1 librados: starting msgr at 2024-03-29T11:24:43.493 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.501+0000 7f7f59e82640 1 librados: starting objecter 2024-03-29T11:24:43.493 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.501+0000 7f7f59e82640 1 Processor -- start 2024-03-29T11:24:43.493 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.501+0000 7f7f59e82640 1 -- start start 2024-03-29T11:24:43.493 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.501+0000 7f7f59e82640 1 librados: setting wanted keys 2024-03-29T11:24:43.493 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.501+0000 7f7f59e82640 1 librados: calling monclient init 2024-03-29T11:24:43.493 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.501+0000 7f7f59e82640 10 monclient: init 2024-03-29T11:24:43.493 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.502+0000 7f7f59e82640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:43.493 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.502+0000 7f7f59e82640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:24:43.494 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.502+0000 7f7f59e82640 10 monclient(hunting): picked mon.noname-b con 0x7f7f540ac5a0 addr v1:172.21.15.119:6789/0 2024-03-29T11:24:43.494 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.502+0000 7f7f59e82640 10 monclient(hunting): picked mon.noname-c con 0x7f7f540b0770 addr v1:172.21.15.104:6790/0 2024-03-29T11:24:43.494 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.502+0000 7f7f59e82640 10 monclient(hunting): picked mon.noname-a con 0x7f7f5414a700 addr v1:172.21.15.104:6789/0 2024-03-29T11:24:43.494 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.502+0000 7f7f59e82640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f54128030 con 0x7f7f5414a700 2024-03-29T11:24:43.494 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.502+0000 7f7f52ffd640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f7f540b0770 0x7f7f541dbb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:48408/0 (socket says 172.21.15.104:48408) 2024-03-29T11:24:43.494 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.502+0000 7f7f52ffd640 1 -- 172.21.15.104:0/1051016139 learned_addr learned my addr 172.21.15.104:0/1051016139 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:24:43.494 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.503+0000 7f7f59e82640 1 -- 172.21.15.104:0/1051016139 --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f54125cb0 con 0x7f7f540ac5a0 2024-03-29T11:24:43.494 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.503+0000 7f7f59e82640 1 -- 172.21.15.104:0/1051016139 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f54145930 con 0x7f7f540b0770 2024-03-29T11:24:43.494 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.503+0000 7f7f59e82640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:43.494 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.503+0000 7f7f59e82640 10 monclient(hunting): authenticate will time out at 1264.754199s 2024-03-29T11:24:43.494 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.503+0000 7f7f3bfff640 10 client.?.objecter ms_handle_connect 0x7f7f5414a700 2024-03-29T11:24:43.494 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.503+0000 7f7f3bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:43.494 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.503+0000 7f7f3bfff640 10 client.?.objecter ms_handle_connect 0x7f7f540b0770 2024-03-29T11:24:43.494 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.503+0000 7f7f3bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:43.495 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.503+0000 7f7f3bfff640 10 client.?.objecter ms_handle_connect 0x7f7f540ac5a0 2024-03-29T11:24:43.495 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.503+0000 7f7f3bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:43.495 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.503+0000 7f7f3bfff640 1 -- 172.21.15.104:0/1051016139 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3438512655 0 0) 0x7f7f54128030 con 0x7f7f5414a700 2024-03-29T11:24:43.495 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.503+0000 7f7f3bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:43.495 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.503+0000 7f7f3bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:43.495 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.503+0000 7f7f3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:43.495 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.503+0000 7f7f3bfff640 10 monclient(hunting): my global_id is 4431 2024-03-29T11:24:43.495 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.503+0000 7f7f3bfff640 1 -- 172.21.15.104:0/1051016139 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f40004eb0 con 0x7f7f5414a700 2024-03-29T11:24:43.495 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.503+0000 7f7f3bfff640 1 -- 172.21.15.104:0/1051016139 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2860341948 0 0) 0x7f7f54145930 con 0x7f7f540b0770 2024-03-29T11:24:43.495 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.503+0000 7f7f3bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:43.495 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.503+0000 7f7f3bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:43.495 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.503+0000 7f7f3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:43.495 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.503+0000 7f7f3bfff640 10 monclient(hunting): my global_id is 4433 2024-03-29T11:24:43.495 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.503+0000 7f7f3bfff640 1 -- 172.21.15.104:0/1051016139 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f54128030 con 0x7f7f540b0770 2024-03-29T11:24:43.495 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.503+0000 7f7f3bfff640 1 -- 172.21.15.104:0/1051016139 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1073732659 0 0) 0x7f7f54125cb0 con 0x7f7f540ac5a0 2024-03-29T11:24:43.497 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.503+0000 7f7f3bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:43.497 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.503+0000 7f7f3bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:43.497 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.503+0000 7f7f3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:43.497 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.503+0000 7f7f3bfff640 10 monclient(hunting): my global_id is 4405 2024-03-29T11:24:43.497 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.503+0000 7f7f3bfff640 1 -- 172.21.15.104:0/1051016139 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f54145930 con 0x7f7f540ac5a0 2024-03-29T11:24:43.497 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.504+0000 7f7f3bfff640 1 -- 172.21.15.104:0/1051016139 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1271498798 0 0) 0x7f7f54145930 con 0x7f7f540ac5a0 2024-03-29T11:24:43.497 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.504+0000 7f7f3bfff640 1 -- 172.21.15.104:0/1051016139 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f54125cb0 con 0x7f7f540ac5a0 2024-03-29T11:24:43.497 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.504+0000 7f7f3bfff640 1 -- 172.21.15.104:0/1051016139 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f7f4c0018b0 con 0x7f7f540ac5a0 2024-03-29T11:24:43.497 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.504+0000 7f7f3bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:43.498 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.504+0000 7f7f3bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:43.498 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.504+0000 7f7f3bfff640 10 monclient(hunting): dump: 2024-03-29T11:24:43.498 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:24:43.498 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:24:43.498 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:43.498 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:43.498 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:43.498 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:43.498 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:24:43.498 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:24:43.499 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:24:43.499 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:43.499 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.504+0000 7f7f3bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:43.499 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.504+0000 7f7f3bfff640 1 -- 172.21.15.104:0/1051016139 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4073807480 0 0) 0x7f7f40004eb0 con 0x7f7f5414a700 2024-03-29T11:24:43.499 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.504+0000 7f7f3bfff640 1 -- 172.21.15.104:0/1051016139 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f54145930 con 0x7f7f5414a700 2024-03-29T11:24:43.499 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.504+0000 7f7f3bfff640 1 -- 172.21.15.104:0/1051016139 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f7f48001b20 con 0x7f7f5414a700 2024-03-29T11:24:43.499 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.504+0000 7f7f3bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:43.499 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.504+0000 7f7f3bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:43.499 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.504+0000 7f7f3bfff640 10 monclient(hunting): dump: 2024-03-29T11:24:43.499 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:24:43.499 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:24:43.499 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:43.499 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:43.499 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:43.499 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:43.500 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:24:43.500 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:24:43.500 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:24:43.500 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:43.500 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.504+0000 7f7f3bfff640 1 -- 172.21.15.104:0/1051016139 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4184549589 0 0) 0x7f7f54128030 con 0x7f7f540b0770 2024-03-29T11:24:43.500 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.504+0000 7f7f3bfff640 1 -- 172.21.15.104:0/1051016139 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f40004eb0 con 0x7f7f540b0770 2024-03-29T11:24:43.500 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.504+0000 7f7f3bfff640 1 -- 172.21.15.104:0/1051016139 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f7f44001b50 con 0x7f7f540b0770 2024-03-29T11:24:43.500 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.504+0000 7f7f3bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:43.500 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.505+0000 7f7f3bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:43.500 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.505+0000 7f7f3bfff640 10 monclient(hunting): dump: 2024-03-29T11:24:43.500 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:24:43.500 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:24:43.500 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:43.501 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:43.501 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:43.501 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:43.501 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:24:43.501 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:24:43.501 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:24:43.501 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:43.501 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.505+0000 7f7f3bfff640 1 -- 172.21.15.104:0/1051016139 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1517185215 0 0) 0x7f7f54125cb0 con 0x7f7f540ac5a0 2024-03-29T11:24:43.501 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.505+0000 7f7f3bfff640 1 -- 172.21.15.104:0/1051016139 >> v1:172.21.15.104:6790/0 conn(0x7f7f540b0770 legacy=0x7f7f541dbb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:43.501 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.505+0000 7f7f3bfff640 1 -- 172.21.15.104:0/1051016139 >> v1:172.21.15.104:6789/0 conn(0x7f7f5414a700 legacy=0x7f7f54144c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:43.501 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.505+0000 7f7f3bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:24:43.501 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.505+0000 7f7f3bfff640 1 monclient: found mon.b 2024-03-29T11:24:43.501 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.505+0000 7f7f3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:43.501 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.505+0000 7f7f3bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:24:43.501 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.505+0000 7f7f3bfff640 1 -- 172.21.15.104:0/1051016139 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f54127190 con 0x7f7f540ac5a0 2024-03-29T11:24:43.502 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.505+0000 7f7f3bfff640 10 monclient: _finish_auth 0 2024-03-29T11:24:43.502 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.505+0000 7f7f3bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:24:43.502 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.505+0000 7f7f3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:43.502 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.505+0000 7f7f59e82640 5 monclient: authenticate success, global_id 4405 2024-03-29T11:24:43.502 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.505+0000 7f7f59e82640 10 monclient: _renew_subs 2024-03-29T11:24:43.502 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.505+0000 7f7f59e82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:24:43.502 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.505+0000 7f7f59e82640 1 -- 172.21.15.104:0/1051016139 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f541239d0 con 0x7f7f540ac5a0 2024-03-29T11:24:43.502 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.505+0000 7f7f59e82640 10 client.4405.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:43.502 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.505+0000 7f7f59e82640 10 monclient: _renew_subs 2024-03-29T11:24:43.502 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.505+0000 7f7f59e82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:24:43.502 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.505+0000 7f7f59e82640 1 -- 172.21.15.104:0/1051016139 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f541dc400 con 0x7f7f540ac5a0 2024-03-29T11:24:43.502 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.505+0000 7f7f59e82640 1 librados: init done 2024-03-29T11:24:43.502 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.505+0000 7f7f3bfff640 1 -- 172.21.15.104:0/1051016139 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f4c001d40 con 0x7f7f540ac5a0 2024-03-29T11:24:43.502 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.506+0000 7f7f3bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:43.503 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.506+0000 7f7f3bfff640 1 -- 172.21.15.104:0/1051016139 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f7f4c0037b0 con 0x7f7f540ac5a0 2024-03-29T11:24:43.503 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.506+0000 7f7f3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:43.503 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.506+0000 7f7f3bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:43.503 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.506+0000 7f7f3bfff640 10 monclient: dump: 2024-03-29T11:24:43.503 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:24:43.503 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:24:43.503 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:43.503 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:43.503 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:43.503 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:43.503 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:24:43.503 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:24:43.503 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:24:43.503 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:43.504 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.506+0000 7f7f51ffb640 4 set_mon_vals no callback set 2024-03-29T11:24:43.504 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.506+0000 7f7f59e82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:43.504 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.508+0000 7f7f3bfff640 1 -- 172.21.15.104:0/1051016139 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f7f4c019780 con 0x7f7f540ac5a0 2024-03-29T11:24:43.504 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.508+0000 7f7f59e82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:43.504 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.508+0000 7f7f59e82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:24:43.504 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.508+0000 7f7f59e82640 1 -- 172.21.15.104:0/1051016139 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f54127320 con 0x7f7f540ac5a0 2024-03-29T11:24:43.504 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.509+0000 7f7f3bfff640 1 -- 172.21.15.104:0/1051016139 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f7f4c0837d0 con 0x7f7f540ac5a0 2024-03-29T11:24:43.504 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.509+0000 7f7f3bfff640 10 client.4405.objecter ms_dispatch 0x7f7f54000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:43.504 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.509+0000 7f7f3bfff640 3 client.4405.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:43.504 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.509+0000 7f7f3bfff640 3 client.4405.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:43.504 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.509+0000 7f7f3bfff640 20 client.4405.objecter dump_active .. 0 homeless 2024-03-29T11:24:43.504 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.509+0000 7f7f52ffd640 10 monclient: get_auth_request con 0x7f7f4006e1c0 auth_method 0 2024-03-29T11:24:43.504 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.509+0000 7f7f3bfff640 10 client.4405.objecter ms_handle_connect 0x7f7f4006e1c0 2024-03-29T11:24:43.505 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.515+0000 7f7f3bfff640 1 -- 172.21.15.104:0/1051016139 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7f4c04f0b0 con 0x7f7f540ac5a0 2024-03-29T11:24:43.505 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.515+0000 7f7f3bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:43.505 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.515+0000 7f7f3bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:43.834 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.851+0000 7f7f59e82640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:43.834 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.851+0000 7f7f59e82640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:43.834 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.851+0000 7f7f59e82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:24:43.834 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.852+0000 7f7f59e82640 1 -- 172.21.15.104:0/1051016139 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7f541db7d0 con 0x7f7f540ac5a0 2024-03-29T11:24:43.837 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.855+0000 7f7f3bfff640 1 -- 172.21.15.104:0/1051016139 <== mon.1 v1:172.21.15.119:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f7f4c050fe0 con 0x7f7f540ac5a0 2024-03-29T11:24:43.837 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.855+0000 7f7f3bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:43.838 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.855+0000 7f7f3bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:43.838 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:24:43.838 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:43.841 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.859+0000 7f7f59e82640 10 librados: watch_flush enter 2024-03-29T11:24:43.841 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.859+0000 7f7f59e82640 10 librados: watch_flush exit 2024-03-29T11:24:43.841 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.859+0000 7f7f59e82640 20 client.4405.objecter shutdown clearing up homeless session... 2024-03-29T11:24:43.841 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.859+0000 7f7f59e82640 10 client.4405.objecter successfully canceled tick 2024-03-29T11:24:43.841 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.859+0000 7f7f59e82640 1 -- 172.21.15.104:0/1051016139 >> v1:172.21.15.104:6816/3341126660 conn(0x7f7f4006e1c0 legacy=0x7f7f40070680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:43.841 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.859+0000 7f7f59e82640 10 monclient: shutdown 2024-03-29T11:24:43.841 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.859+0000 7f7f59e82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:43.842 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.859+0000 7f7f59e82640 1 -- 172.21.15.104:0/1051016139 >> v1:172.21.15.119:6789/0 conn(0x7f7f540ac5a0 legacy=0x7f7f54126a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:43.842 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.860+0000 7f7f59e82640 1 -- 172.21.15.104:0/1051016139 shutdown_connections 2024-03-29T11:24:43.842 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.860+0000 7f7f59e82640 1 -- 172.21.15.104:0/1051016139 >> 172.21.15.104:0/1051016139 conn(0x7f7f5406b660 msgr2=0x7f7f540ae2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:43.843 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.860+0000 7f7f59e82640 1 -- 172.21.15.104:0/1051016139 shutdown_connections 2024-03-29T11:24:43.843 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.860+0000 7f7f59e82640 1 -- 172.21.15.104:0/1051016139 wait complete. 2024-03-29T11:24:43.843 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:43.860+0000 7f7f59e82640 1 librados: shutdown 2024-03-29T11:24:43.857 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:50.664 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:50.816 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.833+0000 7fdadaaae640 10 monclient: get_monmap_and_config 2024-03-29T11:24:50.816 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.833+0000 7fdadaaae640 10 monclient: build_initial_monmap 2024-03-29T11:24:50.816 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.833+0000 7fdadaaae640 10 monclient: monmap: 2024-03-29T11:24:50.816 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:24:50.816 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:24:50.816 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:24:50.834480+0000 2024-03-29T11:24:50.816 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:24:50.834480+0000 2024-03-29T11:24:50.816 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:50.816 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:50.816 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:24:50.817 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:24:50.817 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:24:50.817 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:50.817 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.835+0000 7fdadaaae640 1 Processor -- start 2024-03-29T11:24:50.817 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.835+0000 7fdadaaae640 1 -- start start 2024-03-29T11:24:50.817 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.835+0000 7fdadaaae640 10 monclient: init 2024-03-29T11:24:50.820 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.836+0000 7fdadaaae640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:50.820 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.836+0000 7fdadaaae640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:24:50.820 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.837+0000 7fdadaaae640 10 monclient(hunting): picked mon.noname-b con 0x7fdad406ee40 addr v1:172.21.15.104:6790/0 2024-03-29T11:24:50.820 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.837+0000 7fdadaaae640 10 monclient(hunting): picked mon.noname-a con 0x7fdad40aeda0 addr v1:172.21.15.104:6789/0 2024-03-29T11:24:50.820 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.837+0000 7fdad884a640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7fdad406ee40 0x7fdad413bd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:50146/0 (socket says 172.21.15.104:50146) 2024-03-29T11:24:50.820 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.837+0000 7fdad884a640 1 -- 172.21.15.104:0/582681396 learned_addr learned my addr 172.21.15.104:0/582681396 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:24:50.820 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.837+0000 7fdadaaae640 10 monclient(hunting): picked mon.noname-c con 0x7fdad413c470 addr v1:172.21.15.119:6789/0 2024-03-29T11:24:50.820 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.837+0000 7fdadaaae640 1 -- 172.21.15.104:0/582681396 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdad406fdb0 con 0x7fdad40aeda0 2024-03-29T11:24:50.820 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.837+0000 7fdadaaae640 1 -- 172.21.15.104:0/582681396 --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdad4071090 con 0x7fdad413c470 2024-03-29T11:24:50.821 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.837+0000 7fdadaaae640 1 -- 172.21.15.104:0/582681396 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdad4071370 con 0x7fdad406ee40 2024-03-29T11:24:50.821 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.837+0000 7fdadaaae640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:50.821 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.837+0000 7fdadaaae640 10 monclient(hunting): authenticate will time out at 1272.088535s 2024-03-29T11:24:50.821 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.837+0000 7fdad37fe640 1 -- 172.21.15.104:0/582681396 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1437275173 0 0) 0x7fdad4071090 con 0x7fdad413c470 2024-03-29T11:24:50.821 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.838+0000 7fdad37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:50.821 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.838+0000 7fdad37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:50.821 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.838+0000 7fdad37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:50.821 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.838+0000 7fdad37fe640 10 monclient(hunting): my global_id is 4408 2024-03-29T11:24:50.821 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.838+0000 7fdad37fe640 1 -- 172.21.15.104:0/582681396 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdac4005730 con 0x7fdad413c470 2024-03-29T11:24:50.823 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.838+0000 7fdad37fe640 1 -- 172.21.15.104:0/582681396 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3554615942 0 0) 0x7fdad406fdb0 con 0x7fdad40aeda0 2024-03-29T11:24:50.824 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.838+0000 7fdad37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:50.824 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.838+0000 7fdad37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:50.824 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.838+0000 7fdad37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:50.824 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.838+0000 7fdad37fe640 10 monclient(hunting): my global_id is 4434 2024-03-29T11:24:50.824 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.838+0000 7fdad37fe640 1 -- 172.21.15.104:0/582681396 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdad4071090 con 0x7fdad40aeda0 2024-03-29T11:24:50.824 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.838+0000 7fdad37fe640 1 -- 172.21.15.104:0/582681396 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1917475373 0 0) 0x7fdad4071370 con 0x7fdad406ee40 2024-03-29T11:24:50.824 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.838+0000 7fdad37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:50.824 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.838+0000 7fdad37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:50.824 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.838+0000 7fdad37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:50.824 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.838+0000 7fdad37fe640 10 monclient(hunting): my global_id is 4436 2024-03-29T11:24:50.824 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.838+0000 7fdad37fe640 1 -- 172.21.15.104:0/582681396 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdad406fdb0 con 0x7fdad406ee40 2024-03-29T11:24:50.824 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.838+0000 7fdad37fe640 1 -- 172.21.15.104:0/582681396 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3650578341 0 0) 0x7fdac4005730 con 0x7fdad413c470 2024-03-29T11:24:50.824 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.838+0000 7fdad37fe640 1 -- 172.21.15.104:0/582681396 >> v1:172.21.15.104:6790/0 conn(0x7fdad406ee40 legacy=0x7fdad413bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:50.824 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.838+0000 7fdad37fe640 1 -- 172.21.15.104:0/582681396 >> v1:172.21.15.104:6789/0 conn(0x7fdad40aeda0 legacy=0x7fdad4071790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:50.825 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.838+0000 7fdad37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:24:50.825 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.838+0000 7fdad37fe640 1 monclient: found mon.noname-c 2024-03-29T11:24:50.825 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.838+0000 7fdad37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:50.825 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.838+0000 7fdad37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.119:6789/0 2024-03-29T11:24:50.825 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.838+0000 7fdad37fe640 1 -- 172.21.15.104:0/582681396 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdad4070220 con 0x7fdad413c470 2024-03-29T11:24:50.825 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.838+0000 7fdad37fe640 10 monclient: _finish_auth 0 2024-03-29T11:24:50.825 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.838+0000 7fdad37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:24:50.825 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.838+0000 7fdad37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:50.825 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.838+0000 7fdad37fe640 1 -- 172.21.15.104:0/582681396 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fdacc001a10 con 0x7fdad413c470 2024-03-29T11:24:50.825 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.838+0000 7fdad37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:50.825 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.838+0000 7fdad37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:50.825 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.838+0000 7fdad37fe640 10 monclient: dump: 2024-03-29T11:24:50.825 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:24:50.825 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:24:50.826 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:50.826 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:50.826 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:50.826 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:50.826 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:24:50.826 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:24:50.826 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:24:50.826 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:50.826 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.838+0000 7fdadaaae640 5 monclient: authenticate success, global_id 4408 2024-03-29T11:24:50.826 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.838+0000 7fdadaaae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:50.827 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.839+0000 7fdad37fe640 1 -- 172.21.15.104:0/582681396 <== mon.1 v1:172.21.15.119:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdacc001b70 con 0x7fdad413c470 2024-03-29T11:24:50.827 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.839+0000 7fdad37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:50.827 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.839+0000 7fdad37fe640 1 -- 172.21.15.104:0/582681396 <== mon.1 v1:172.21.15.119:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fdacc001f80 con 0x7fdad413c470 2024-03-29T11:24:50.827 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.839+0000 7fdad37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:50.827 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.839+0000 7fdad37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:50.827 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.839+0000 7fdad37fe640 10 monclient: dump: 2024-03-29T11:24:50.827 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:24:50.827 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:24:50.827 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:50.827 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:50.827 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:50.827 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:50.827 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:24:50.827 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:24:50.827 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:24:50.828 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:50.828 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.839+0000 7fdadaaae640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:50.828 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.839+0000 7fdadaaae640 4 set_mon_vals no callback set 2024-03-29T11:24:50.828 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.840+0000 7fdadaaae640 10 monclient: shutdown 2024-03-29T11:24:50.828 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.840+0000 7fdadaaae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:50.828 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.840+0000 7fdadaaae640 1 -- 172.21.15.104:0/582681396 >> v1:172.21.15.119:6789/0 conn(0x7fdad413c470 legacy=0x7fdad414a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:50.828 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.840+0000 7fdadaaae640 1 -- 172.21.15.104:0/582681396 shutdown_connections 2024-03-29T11:24:50.828 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.840+0000 7fdadaaae640 1 -- 172.21.15.104:0/582681396 >> 172.21.15.104:0/582681396 conn(0x7fdad4067470 msgr2=0x7fdad40b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:50.828 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.841+0000 7fdadaaae640 1 -- 172.21.15.104:0/582681396 shutdown_connections 2024-03-29T11:24:50.828 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.841+0000 7fdadaaae640 1 -- 172.21.15.104:0/582681396 wait complete. 2024-03-29T11:24:50.833 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.842+0000 7fdadaaae640 10 monclient: build_initial_monmap 2024-03-29T11:24:50.834 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.842+0000 7fdadaaae640 10 monclient: monmap: 2024-03-29T11:24:50.834 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:24:50.834 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:50.834 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:24:50.834 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:24:50.834 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:50.834 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:50.834 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:24:50.834 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:24:50.834 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:24:50.834 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:50.834 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.843+0000 7fdadaaae640 1 librados: starting msgr at 2024-03-29T11:24:50.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.843+0000 7fdadaaae640 1 librados: starting objecter 2024-03-29T11:24:50.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.843+0000 7fdadaaae640 1 Processor -- start 2024-03-29T11:24:50.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.843+0000 7fdadaaae640 1 -- start start 2024-03-29T11:24:50.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.843+0000 7fdadaaae640 1 librados: setting wanted keys 2024-03-29T11:24:50.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.843+0000 7fdadaaae640 1 librados: calling monclient init 2024-03-29T11:24:50.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.843+0000 7fdadaaae640 10 monclient: init 2024-03-29T11:24:50.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.844+0000 7fdadaaae640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:50.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.844+0000 7fdadaaae640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:24:50.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.844+0000 7fdadaaae640 10 monclient(hunting): picked mon.noname-b con 0x7fdad406ee40 addr v1:172.21.15.119:6789/0 2024-03-29T11:24:50.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.844+0000 7fdadaaae640 10 monclient(hunting): picked mon.noname-c con 0x7fdad40aeda0 addr v1:172.21.15.104:6790/0 2024-03-29T11:24:50.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.844+0000 7fdadaaae640 10 monclient(hunting): picked mon.noname-a con 0x7fdad413c470 addr v1:172.21.15.104:6789/0 2024-03-29T11:24:50.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.845+0000 7fdadaaae640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdad40ac540 con 0x7fdad413c470 2024-03-29T11:24:50.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.845+0000 7fdadaaae640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdad413e0b0 con 0x7fdad406ee40 2024-03-29T11:24:50.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.845+0000 7fdadaaae640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdad40ab1b0 con 0x7fdad40aeda0 2024-03-29T11:24:50.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.845+0000 7fdadaaae640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:50.836 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.845+0000 7fdadaaae640 10 monclient(hunting): authenticate will time out at 1272.096313s 2024-03-29T11:24:50.836 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.845+0000 7fdad884a640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fdad406ee40 0x7fdad40aaaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:52972/0 (socket says 172.21.15.104:52972) 2024-03-29T11:24:50.836 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.845+0000 7fdad884a640 1 -- 172.21.15.104:0/394289028 learned_addr learned my addr 172.21.15.104:0/394289028 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:24:50.836 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.845+0000 7fdad17fa640 10 client.?.objecter ms_handle_connect 0x7fdad406ee40 2024-03-29T11:24:50.836 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.845+0000 7fdad17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:50.836 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.845+0000 7fdad17fa640 10 client.?.objecter ms_handle_connect 0x7fdad40aeda0 2024-03-29T11:24:50.836 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.845+0000 7fdad17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:50.836 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.846+0000 7fdad17fa640 10 client.?.objecter ms_handle_connect 0x7fdad413c470 2024-03-29T11:24:50.836 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.846+0000 7fdad17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:50.836 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.846+0000 7fdad17fa640 1 -- 172.21.15.104:0/394289028 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3593820206 0 0) 0x7fdad413e0b0 con 0x7fdad406ee40 2024-03-29T11:24:50.836 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.846+0000 7fdad17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:50.836 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.846+0000 7fdad17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:50.836 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.846+0000 7fdad17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:50.836 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.846+0000 7fdad17fa640 10 monclient(hunting): my global_id is 4411 2024-03-29T11:24:50.837 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.846+0000 7fdad17fa640 1 -- 172.21.15.104:0/394289028 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdab8004eb0 con 0x7fdad406ee40 2024-03-29T11:24:50.837 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.846+0000 7fdad17fa640 1 -- 172.21.15.104:0/394289028 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 94313333 0 0) 0x7fdad40ac540 con 0x7fdad413c470 2024-03-29T11:24:50.837 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.846+0000 7fdad17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:50.837 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.846+0000 7fdad17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:50.837 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.846+0000 7fdad17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:50.837 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.846+0000 7fdad17fa640 10 monclient(hunting): my global_id is 4437 2024-03-29T11:24:50.837 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.846+0000 7fdad17fa640 1 -- 172.21.15.104:0/394289028 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdad413e0b0 con 0x7fdad413c470 2024-03-29T11:24:50.837 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.846+0000 7fdad17fa640 1 -- 172.21.15.104:0/394289028 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1504450419 0 0) 0x7fdad40ab1b0 con 0x7fdad40aeda0 2024-03-29T11:24:50.837 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.846+0000 7fdad17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:50.837 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.846+0000 7fdad17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:50.837 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.846+0000 7fdad17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:50.837 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.846+0000 7fdad17fa640 10 monclient(hunting): my global_id is 4439 2024-03-29T11:24:50.837 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.846+0000 7fdad17fa640 1 -- 172.21.15.104:0/394289028 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdad40ac540 con 0x7fdad40aeda0 2024-03-29T11:24:50.837 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.846+0000 7fdad17fa640 1 -- 172.21.15.104:0/394289028 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1812224926 0 0) 0x7fdab8004eb0 con 0x7fdad406ee40 2024-03-29T11:24:50.837 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.846+0000 7fdad17fa640 1 -- 172.21.15.104:0/394289028 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdad40ab1b0 con 0x7fdad406ee40 2024-03-29T11:24:50.838 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.847+0000 7fdad17fa640 1 -- 172.21.15.104:0/394289028 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fdac80019f0 con 0x7fdad406ee40 2024-03-29T11:24:50.838 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.847+0000 7fdad17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:50.838 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.847+0000 7fdad17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:50.838 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.847+0000 7fdad17fa640 10 monclient(hunting): dump: 2024-03-29T11:24:50.838 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:24:50.838 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:24:50.838 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:50.838 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:50.838 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:50.838 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:50.838 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:24:50.839 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:24:50.839 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:24:50.839 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:50.839 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.847+0000 7fdad17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:50.839 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.847+0000 7fdad17fa640 1 -- 172.21.15.104:0/394289028 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 10770036 0 0) 0x7fdad413e0b0 con 0x7fdad413c470 2024-03-29T11:24:50.839 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.847+0000 7fdad17fa640 1 -- 172.21.15.104:0/394289028 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdab8004eb0 con 0x7fdad413c470 2024-03-29T11:24:50.839 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.847+0000 7fdad17fa640 1 -- 172.21.15.104:0/394289028 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fdacc001e80 con 0x7fdad413c470 2024-03-29T11:24:50.839 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.847+0000 7fdad17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:50.839 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.847+0000 7fdad17fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:50.839 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.847+0000 7fdad17fa640 10 monclient(hunting): dump: 2024-03-29T11:24:50.839 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:24:50.839 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:24:50.839 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:50.839 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:50.839 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:50.839 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:50.840 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:24:50.840 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:24:50.840 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:24:50.840 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:50.840 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.847+0000 7fdad17fa640 1 -- 172.21.15.104:0/394289028 <== mon.0 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 506400943 0 0) 0x7fdab8004eb0 con 0x7fdad413c470 2024-03-29T11:24:50.840 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.847+0000 7fdad17fa640 1 -- 172.21.15.104:0/394289028 >> v1:172.21.15.104:6790/0 conn(0x7fdad40aeda0 legacy=0x7fdad40ac9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:50.840 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.847+0000 7fdad17fa640 1 -- 172.21.15.104:0/394289028 >> v1:172.21.15.119:6789/0 conn(0x7fdad406ee40 legacy=0x7fdad40aaaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:50.840 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.847+0000 7fdad17fa640 10 monclient: _finish_hunting 0 2024-03-29T11:24:50.840 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.847+0000 7fdad17fa640 1 monclient: found mon.a 2024-03-29T11:24:50.840 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.847+0000 7fdad17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:50.840 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.847+0000 7fdad17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:24:50.840 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.847+0000 7fdad17fa640 1 -- 172.21.15.104:0/394289028 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdad40ab340 con 0x7fdad413c470 2024-03-29T11:24:50.840 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.847+0000 7fdad17fa640 10 monclient: _finish_auth 0 2024-03-29T11:24:50.840 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.847+0000 7fdad17fa640 10 monclient: _check_auth_tickets 2024-03-29T11:24:50.840 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.847+0000 7fdad17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:50.841 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.847+0000 7fdadaaae640 5 monclient: authenticate success, global_id 4437 2024-03-29T11:24:50.841 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.847+0000 7fdadaaae640 10 monclient: _renew_subs 2024-03-29T11:24:50.841 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.847+0000 7fdadaaae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:24:50.841 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.847+0000 7fdadaaae640 1 -- 172.21.15.104:0/394289028 --> v1:172.21.15.104:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdad4139390 con 0x7fdad413c470 2024-03-29T11:24:50.841 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.847+0000 7fdadaaae640 10 client.4437.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:50.841 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.848+0000 7fdadaaae640 10 monclient: _renew_subs 2024-03-29T11:24:50.841 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.848+0000 7fdadaaae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:24:50.841 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.848+0000 7fdadaaae640 1 -- 172.21.15.104:0/394289028 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdad40ad7f0 con 0x7fdad413c470 2024-03-29T11:24:50.841 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.848+0000 7fdadaaae640 1 librados: init done 2024-03-29T11:24:50.841 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.848+0000 7fdad17fa640 1 -- 172.21.15.104:0/394289028 <== mon.0 v1:172.21.15.104:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdacc001fe0 con 0x7fdad413c470 2024-03-29T11:24:50.841 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.848+0000 7fdad17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:50.841 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.848+0000 7fdad17fa640 1 -- 172.21.15.104:0/394289028 <== mon.0 v1:172.21.15.104:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fdacc003dc0 con 0x7fdad413c470 2024-03-29T11:24:50.841 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.848+0000 7fdad2ffd640 4 set_mon_vals no callback set 2024-03-29T11:24:50.841 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.848+0000 7fdad17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:50.841 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.848+0000 7fdad17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:50.841 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.848+0000 7fdad17fa640 10 monclient: dump: 2024-03-29T11:24:50.842 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:24:50.842 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:24:50.842 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:50.842 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:50.842 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:50.842 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:50.842 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:24:50.842 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:24:50.842 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:24:50.842 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:50.842 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.849+0000 7fdadaaae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:50.842 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.849+0000 7fdadaaae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:50.842 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.849+0000 7fdadaaae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:24:50.842 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.849+0000 7fdadaaae640 1 -- 172.21.15.104:0/394289028 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdab4005080 con 0x7fdad413c470 2024-03-29T11:24:50.843 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.851+0000 7fdad17fa640 1 -- 172.21.15.104:0/394289028 <== mon.0 v1:172.21.15.104:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7fdacc019d90 con 0x7fdad413c470 2024-03-29T11:24:50.843 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.852+0000 7fdad884a640 10 monclient: get_auth_request con 0x7fdab806dd00 auth_method 0 2024-03-29T11:24:50.844 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.852+0000 7fdad17fa640 1 -- 172.21.15.104:0/394289028 <== mon.0 v1:172.21.15.104:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7fdacc083ae0 con 0x7fdad413c470 2024-03-29T11:24:50.844 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.852+0000 7fdad17fa640 10 client.4437.objecter ms_dispatch 0x7fdad4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:50.844 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.852+0000 7fdad17fa640 3 client.4437.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:50.844 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.852+0000 7fdad17fa640 3 client.4437.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:50.844 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.852+0000 7fdad17fa640 20 client.4437.objecter dump_active .. 0 homeless 2024-03-29T11:24:50.844 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.852+0000 7fdad17fa640 10 client.4437.objecter ms_handle_connect 0x7fdab806dd00 2024-03-29T11:24:50.847 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.865+0000 7fdad17fa640 1 -- 172.21.15.104:0/394289028 <== mon.0 v1:172.21.15.104:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdacc04f3c0 con 0x7fdad413c470 2024-03-29T11:24:50.847 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.865+0000 7fdad17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:50.847 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:50.865+0000 7fdad17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:51.221 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:51.238+0000 7fdadaaae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:51.221 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:51.238+0000 7fdadaaae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:51.221 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:51.238+0000 7fdadaaae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:24:51.222 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:51.238+0000 7fdadaaae640 1 -- 172.21.15.104:0/394289028 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdab4005330 con 0x7fdad413c470 2024-03-29T11:24:51.223 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:51.240+0000 7fdad17fa640 1 -- 172.21.15.104:0/394289028 <== mon.0 v1:172.21.15.104:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7fdacc0512f0 con 0x7fdad413c470 2024-03-29T11:24:51.223 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:51.240+0000 7fdad17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:51.223 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:51.240+0000 7fdad17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:51.223 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:24:51.224 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:51.227 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:51.245+0000 7fdadaaae640 10 librados: watch_flush enter 2024-03-29T11:24:51.227 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:51.245+0000 7fdadaaae640 10 librados: watch_flush exit 2024-03-29T11:24:51.227 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:51.245+0000 7fdadaaae640 20 client.4437.objecter shutdown clearing up homeless session... 2024-03-29T11:24:51.227 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:51.245+0000 7fdadaaae640 10 client.4437.objecter successfully canceled tick 2024-03-29T11:24:51.228 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:51.245+0000 7fdadaaae640 1 -- 172.21.15.104:0/394289028 >> v1:172.21.15.104:6816/3341126660 conn(0x7fdab806dd00 legacy=0x7fdab80701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:51.228 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:51.245+0000 7fdadaaae640 10 monclient: shutdown 2024-03-29T11:24:51.228 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:51.245+0000 7fdadaaae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:51.228 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:51.245+0000 7fdadaaae640 1 -- 172.21.15.104:0/394289028 >> v1:172.21.15.104:6789/0 conn(0x7fdad413c470 legacy=0x7fdad40ad0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:51.228 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:51.245+0000 7fdadaaae640 1 -- 172.21.15.104:0/394289028 shutdown_connections 2024-03-29T11:24:51.229 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:51.245+0000 7fdadaaae640 1 -- 172.21.15.104:0/394289028 >> 172.21.15.104:0/394289028 conn(0x7fdad4067470 msgr2=0x7fdad413bb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:51.229 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:51.246+0000 7fdadaaae640 1 -- 172.21.15.104:0/394289028 shutdown_connections 2024-03-29T11:24:51.229 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:51.246+0000 7fdadaaae640 1 -- 172.21.15.104:0/394289028 wait complete. 2024-03-29T11:24:51.229 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:51.246+0000 7fdadaaae640 1 librados: shutdown 2024-03-29T11:24:51.242 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:58.049 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:58.206 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.222+0000 7f8279686640 10 monclient: get_monmap_and_config 2024-03-29T11:24:58.206 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.222+0000 7f8279686640 10 monclient: build_initial_monmap 2024-03-29T11:24:58.206 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.223+0000 7f8279686640 10 monclient: monmap: 2024-03-29T11:24:58.206 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:24:58.206 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:24:58.206 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:24:58.223924+0000 2024-03-29T11:24:58.206 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:24:58.223924+0000 2024-03-29T11:24:58.206 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:58.206 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:58.206 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:24:58.206 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:24:58.207 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:24:58.207 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:58.207 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.224+0000 7f8279686640 1 Processor -- start 2024-03-29T11:24:58.207 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.224+0000 7f8279686640 1 -- start start 2024-03-29T11:24:58.207 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.224+0000 7f8279686640 10 monclient: init 2024-03-29T11:24:58.211 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.225+0000 7f8279686640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:58.211 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.225+0000 7f8279686640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:24:58.211 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.225+0000 7f8279686640 10 monclient(hunting): picked mon.noname-c con 0x7f82740b0770 addr v1:172.21.15.119:6789/0 2024-03-29T11:24:58.211 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.225+0000 7f8279686640 10 monclient(hunting): picked mon.noname-a con 0x7f827414a700 addr v1:172.21.15.104:6789/0 2024-03-29T11:24:58.212 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.225+0000 7f8279686640 10 monclient(hunting): picked mon.noname-b con 0x7f82740ac5a0 addr v1:172.21.15.104:6790/0 2024-03-29T11:24:58.212 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.225+0000 7f8279686640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82740aba70 con 0x7f827414a700 2024-03-29T11:24:58.212 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.225+0000 7f8279686640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82740aecb0 con 0x7f82740b0770 2024-03-29T11:24:58.212 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.225+0000 7f8279686640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82740af150 con 0x7f82740ac5a0 2024-03-29T11:24:58.212 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.225+0000 7f8279686640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:58.212 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.225+0000 7f8279686640 10 monclient(hunting): authenticate will time out at 1279.476119s 2024-03-29T11:24:58.212 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.228+0000 7f8272ffd640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f82740b0770 0x7f8274149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:52163/0 (socket says 172.21.15.104:52163) 2024-03-29T11:24:58.212 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.228+0000 7f8272ffd640 1 -- 172.21.15.104:0/1408674160 learned_addr learned my addr 172.21.15.104:0/1408674160 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:24:58.214 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.231+0000 7f8271ffb640 1 -- 172.21.15.104:0/1408674160 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 705087039 0 0) 0x7f82740aba70 con 0x7f827414a700 2024-03-29T11:24:58.214 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.231+0000 7f8271ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:24:58.214 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.231+0000 7f8271ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:58.214 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.231+0000 7f8271ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:58.214 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.231+0000 7f8271ffb640 10 monclient(hunting): my global_id is 4440 2024-03-29T11:24:58.215 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.231+0000 7f8271ffb640 1 -- 172.21.15.104:0/1408674160 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f825c003520 con 0x7f827414a700 2024-03-29T11:24:58.215 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.231+0000 7f8271ffb640 1 -- 172.21.15.104:0/1408674160 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 363762329 0 0) 0x7f82740aecb0 con 0x7f82740b0770 2024-03-29T11:24:58.215 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.231+0000 7f8271ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:24:58.215 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.231+0000 7f8271ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:58.215 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.231+0000 7f8271ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:58.215 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.231+0000 7f8271ffb640 10 monclient(hunting): my global_id is 4414 2024-03-29T11:24:58.215 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.232+0000 7f8271ffb640 1 -- 172.21.15.104:0/1408674160 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82740aba70 con 0x7f82740b0770 2024-03-29T11:24:58.218 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.234+0000 7f8271ffb640 1 -- 172.21.15.104:0/1408674160 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4114377048 0 0) 0x7f82740af150 con 0x7f82740ac5a0 2024-03-29T11:24:58.218 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.234+0000 7f8271ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:24:58.218 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.234+0000 7f8271ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:58.218 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.234+0000 7f8271ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:58.219 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.234+0000 7f8271ffb640 10 monclient(hunting): my global_id is 4442 2024-03-29T11:24:58.219 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.234+0000 7f8271ffb640 1 -- 172.21.15.104:0/1408674160 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82740aecb0 con 0x7f82740ac5a0 2024-03-29T11:24:58.219 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.234+0000 7f8271ffb640 1 -- 172.21.15.104:0/1408674160 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1337571996 0 0) 0x7f82740aba70 con 0x7f82740b0770 2024-03-29T11:24:58.219 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.234+0000 7f8271ffb640 1 -- 172.21.15.104:0/1408674160 >> v1:172.21.15.104:6790/0 conn(0x7f82740ac5a0 legacy=0x7f8274151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:58.219 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.234+0000 7f8271ffb640 1 -- 172.21.15.104:0/1408674160 >> v1:172.21.15.104:6789/0 conn(0x7f827414a700 legacy=0x7f82740abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:58.219 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.234+0000 7f8271ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:24:58.219 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.234+0000 7f8271ffb640 1 monclient: found mon.noname-c 2024-03-29T11:24:58.219 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.234+0000 7f8271ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:58.219 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.234+0000 7f8271ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.119:6789/0 2024-03-29T11:24:58.219 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.234+0000 7f8271ffb640 1 -- 172.21.15.104:0/1408674160 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82740ac9d0 con 0x7f82740b0770 2024-03-29T11:24:58.219 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.234+0000 7f8271ffb640 10 monclient: _finish_auth 0 2024-03-29T11:24:58.219 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.234+0000 7f8271ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:24:58.219 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.234+0000 7f8271ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:58.219 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.234+0000 7f8271ffb640 1 -- 172.21.15.104:0/1408674160 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f826c001930 con 0x7f82740b0770 2024-03-29T11:24:58.219 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.234+0000 7f8271ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:58.220 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.234+0000 7f8271ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:58.220 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.234+0000 7f8271ffb640 10 monclient: dump: 2024-03-29T11:24:58.220 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:24:58.220 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:24:58.220 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:58.220 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:58.220 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:58.220 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:58.220 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:24:58.220 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:24:58.221 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:24:58.221 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:58.221 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.235+0000 7f8271ffb640 1 -- 172.21.15.104:0/1408674160 <== mon.1 v1:172.21.15.119:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f826c001a90 con 0x7f82740b0770 2024-03-29T11:24:58.221 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.235+0000 7f8271ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:58.221 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.235+0000 7f8271ffb640 1 -- 172.21.15.104:0/1408674160 <== mon.1 v1:172.21.15.119:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f826c001e60 con 0x7f82740b0770 2024-03-29T11:24:58.221 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.235+0000 7f8271ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:58.221 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.235+0000 7f8271ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:58.221 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.235+0000 7f8271ffb640 10 monclient: dump: 2024-03-29T11:24:58.221 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:24:58.221 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:24:58.221 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:58.221 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:58.221 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:58.221 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:58.221 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:24:58.221 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:24:58.222 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:24:58.222 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:58.223 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.237+0000 7f8279686640 5 monclient: authenticate success, global_id 4414 2024-03-29T11:24:58.223 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.237+0000 7f8279686640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:58.223 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.237+0000 7f8279686640 4 set_mon_vals no callback set 2024-03-29T11:24:58.223 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.238+0000 7f8279686640 10 monclient: shutdown 2024-03-29T11:24:58.223 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.238+0000 7f8279686640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:58.223 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.238+0000 7f8279686640 1 -- 172.21.15.104:0/1408674160 >> v1:172.21.15.119:6789/0 conn(0x7f82740b0770 legacy=0x7f8274149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:58.223 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.238+0000 7f8279686640 1 -- 172.21.15.104:0/1408674160 shutdown_connections 2024-03-29T11:24:58.223 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.238+0000 7f8279686640 1 -- 172.21.15.104:0/1408674160 >> 172.21.15.104:0/1408674160 conn(0x7f827406b660 msgr2=0x7f827406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:58.224 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.239+0000 7f8279686640 1 -- 172.21.15.104:0/1408674160 shutdown_connections 2024-03-29T11:24:58.225 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.239+0000 7f8279686640 1 -- 172.21.15.104:0/1408674160 wait complete. 2024-03-29T11:24:58.225 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.240+0000 7f8279686640 10 monclient: build_initial_monmap 2024-03-29T11:24:58.226 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.240+0000 7f8279686640 10 monclient: monmap: 2024-03-29T11:24:58.226 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:24:58.226 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:58.226 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:24:58.226 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:24:58.226 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:58.226 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:58.226 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:24:58.226 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:24:58.226 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:24:58.226 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:58.227 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.240+0000 7f8279686640 1 librados: starting msgr at 2024-03-29T11:24:58.227 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.240+0000 7f8279686640 1 librados: starting objecter 2024-03-29T11:24:58.227 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.240+0000 7f8279686640 1 Processor -- start 2024-03-29T11:24:58.227 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.244+0000 7f8279686640 1 -- start start 2024-03-29T11:24:58.227 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.244+0000 7f8279686640 1 librados: setting wanted keys 2024-03-29T11:24:58.227 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.244+0000 7f8279686640 1 librados: calling monclient init 2024-03-29T11:24:58.227 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.244+0000 7f8279686640 10 monclient: init 2024-03-29T11:24:58.228 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.245+0000 7f8279686640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:58.228 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.245+0000 7f8279686640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:24:58.228 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.245+0000 7f8279686640 10 monclient(hunting): picked mon.noname-b con 0x7f82740ac5a0 addr v1:172.21.15.119:6789/0 2024-03-29T11:24:58.228 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.245+0000 7f8279686640 10 monclient(hunting): picked mon.noname-a con 0x7f82740b0770 addr v1:172.21.15.104:6789/0 2024-03-29T11:24:58.228 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.245+0000 7f8279686640 10 monclient(hunting): picked mon.noname-c con 0x7f827414a700 addr v1:172.21.15.104:6790/0 2024-03-29T11:24:58.228 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.245+0000 7f8279686640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82741285a0 con 0x7f82740b0770 2024-03-29T11:24:58.228 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.245+0000 7f8279686640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8274126140 con 0x7f82740ac5a0 2024-03-29T11:24:58.228 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.245+0000 7f8279686640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8274126520 con 0x7f827414a700 2024-03-29T11:24:58.228 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.245+0000 7f8279686640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:58.228 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.245+0000 7f8279686640 10 monclient(hunting): authenticate will time out at 1279.496459s 2024-03-29T11:24:58.233 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.250+0000 7f82727fc640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f82740b0770 0x7f8274145120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:43341/0 (socket says 172.21.15.104:43341) 2024-03-29T11:24:58.233 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.250+0000 7f82727fc640 1 -- 172.21.15.104:0/1624767182 learned_addr learned my addr 172.21.15.104:0/1624767182 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:24:58.233 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.250+0000 7f825b7fe640 10 client.?.objecter ms_handle_connect 0x7f827414a700 2024-03-29T11:24:58.233 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.250+0000 7f825b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:58.233 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.251+0000 7f825b7fe640 10 client.?.objecter ms_handle_connect 0x7f82740b0770 2024-03-29T11:24:58.234 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.251+0000 7f825b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:58.235 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.252+0000 7f825b7fe640 1 -- 172.21.15.104:0/1624767182 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 430635586 0 0) 0x7f8274126520 con 0x7f827414a700 2024-03-29T11:24:58.235 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.252+0000 7f825b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:58.235 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.252+0000 7f825b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:58.235 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.252+0000 7f825b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:58.235 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.252+0000 7f825b7fe640 10 monclient(hunting): my global_id is 4445 2024-03-29T11:24:58.235 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.252+0000 7f825b7fe640 1 -- 172.21.15.104:0/1624767182 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8260004eb0 con 0x7f827414a700 2024-03-29T11:24:58.235 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.252+0000 7f825b7fe640 10 client.?.objecter ms_handle_connect 0x7f82740ac5a0 2024-03-29T11:24:58.235 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.252+0000 7f825b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:58.236 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.252+0000 7f825b7fe640 1 -- 172.21.15.104:0/1624767182 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2790773701 0 0) 0x7f8260004eb0 con 0x7f827414a700 2024-03-29T11:24:58.236 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.252+0000 7f825b7fe640 1 -- 172.21.15.104:0/1624767182 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8274126520 con 0x7f827414a700 2024-03-29T11:24:58.236 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.252+0000 7f825b7fe640 1 -- 172.21.15.104:0/1624767182 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f8264001930 con 0x7f827414a700 2024-03-29T11:24:58.236 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.252+0000 7f825b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:58.236 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.252+0000 7f825b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:58.236 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.252+0000 7f825b7fe640 10 monclient(hunting): dump: 2024-03-29T11:24:58.236 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:24:58.236 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:24:58.236 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:58.236 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:58.237 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:58.237 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:58.237 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:24:58.237 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:24:58.237 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:24:58.237 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:58.237 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.252+0000 7f825b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:58.237 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.253+0000 7f825b7fe640 1 -- 172.21.15.104:0/1624767182 <== mon.2 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3830871464 0 0) 0x7f8274126520 con 0x7f827414a700 2024-03-29T11:24:58.237 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.253+0000 7f825b7fe640 1 -- 172.21.15.104:0/1624767182 >> v1:172.21.15.119:6789/0 conn(0x7f82740ac5a0 legacy=0x7f8274126a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:58.237 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.253+0000 7f825b7fe640 1 -- 172.21.15.104:0/1624767182 >> v1:172.21.15.104:6789/0 conn(0x7f82740b0770 legacy=0x7f8274145120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:58.237 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.253+0000 7f825b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:24:58.237 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.253+0000 7f825b7fe640 1 monclient: found mon.c 2024-03-29T11:24:58.237 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.253+0000 7f825b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:58.237 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.253+0000 7f825b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:24:58.237 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.253+0000 7f825b7fe640 1 -- 172.21.15.104:0/1624767182 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82741275a0 con 0x7f827414a700 2024-03-29T11:24:58.238 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.253+0000 7f825b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:24:58.238 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.253+0000 7f825b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:24:58.238 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.253+0000 7f825b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:58.238 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.253+0000 7f825b7fe640 1 -- 172.21.15.104:0/1624767182 <== mon.2 v1:172.21.15.104:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8264003610 con 0x7f827414a700 2024-03-29T11:24:58.239 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.253+0000 7f825b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:58.239 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.253+0000 7f825b7fe640 1 -- 172.21.15.104:0/1624767182 <== mon.2 v1:172.21.15.104:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f8264003920 con 0x7f827414a700 2024-03-29T11:24:58.239 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.253+0000 7f825b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:58.239 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.253+0000 7f825b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:58.239 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.253+0000 7f825b7fe640 10 monclient: dump: 2024-03-29T11:24:58.239 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:24:58.239 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:24:58.239 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:58.239 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:24:58.239 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:58.239 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:24:58.239 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:24:58.239 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:24:58.239 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:24:58.239 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:24:58.240 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.253+0000 7f82717fa640 4 set_mon_vals no callback set 2024-03-29T11:24:58.240 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.254+0000 7f8279686640 5 monclient: authenticate success, global_id 4445 2024-03-29T11:24:58.240 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.254+0000 7f8279686640 10 monclient: _renew_subs 2024-03-29T11:24:58.240 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.254+0000 7f8279686640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:24:58.240 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.254+0000 7f8279686640 1 -- 172.21.15.104:0/1624767182 --> v1:172.21.15.104:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82741239d0 con 0x7f827414a700 2024-03-29T11:24:58.240 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.254+0000 7f8279686640 10 client.4445.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:58.240 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.254+0000 7f8279686640 10 monclient: _renew_subs 2024-03-29T11:24:58.240 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.254+0000 7f8279686640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:24:58.240 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.254+0000 7f8279686640 1 -- 172.21.15.104:0/1624767182 --> v1:172.21.15.104:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8274127330 con 0x7f827414a700 2024-03-29T11:24:58.240 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.254+0000 7f8279686640 1 librados: init done 2024-03-29T11:24:58.240 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.256+0000 7f8279686640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:58.240 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.256+0000 7f8279686640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:58.240 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.256+0000 7f8279686640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:24:58.240 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.256+0000 7f8279686640 1 -- 172.21.15.104:0/1624767182 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82740ab560 con 0x7f827414a700 2024-03-29T11:24:58.242 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.259+0000 7f825b7fe640 1 -- 172.21.15.104:0/1624767182 <== mon.2 v1:172.21.15.104:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f82640198f0 con 0x7f827414a700 2024-03-29T11:24:58.245 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.262+0000 7f825b7fe640 1 -- 172.21.15.104:0/1624767182 <== mon.2 v1:172.21.15.104:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f8264083750 con 0x7f827414a700 2024-03-29T11:24:58.245 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.262+0000 7f825b7fe640 10 client.4445.objecter ms_dispatch 0x7f8274000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:58.245 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.262+0000 7f825b7fe640 3 client.4445.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:58.245 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.262+0000 7f825b7fe640 3 client.4445.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:58.245 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.262+0000 7f825b7fe640 20 client.4445.objecter dump_active .. 0 homeless 2024-03-29T11:24:58.245 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.262+0000 7f8272ffd640 10 monclient: get_auth_request con 0x7f826006dd00 auth_method 0 2024-03-29T11:24:58.245 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.262+0000 7f825b7fe640 10 client.4445.objecter ms_handle_connect 0x7f826006dd00 2024-03-29T11:24:58.248 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.266+0000 7f825b7fe640 1 -- 172.21.15.104:0/1624767182 <== mon.2 v1:172.21.15.104:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f826404f090 con 0x7f827414a700 2024-03-29T11:24:58.248 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.266+0000 7f825b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:58.249 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.266+0000 7f825b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:58.575 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.590+0000 7f8279686640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:58.575 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.590+0000 7f8279686640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:58.575 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.590+0000 7f8279686640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:24:58.575 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.590+0000 7f8279686640 1 -- 172.21.15.104:0/1624767182 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82741266b0 con 0x7f827414a700 2024-03-29T11:24:58.576 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.593+0000 7f825b7fe640 1 -- 172.21.15.104:0/1624767182 <== mon.2 v1:172.21.15.104:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f8264050fc0 con 0x7f827414a700 2024-03-29T11:24:58.576 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.593+0000 7f825b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:58.576 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.593+0000 7f825b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:58.577 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:24:58.577 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:58.583 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.600+0000 7f8279686640 10 librados: watch_flush enter 2024-03-29T11:24:58.583 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.600+0000 7f8279686640 10 librados: watch_flush exit 2024-03-29T11:24:58.583 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.600+0000 7f8279686640 20 client.4445.objecter shutdown clearing up homeless session... 2024-03-29T11:24:58.583 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.600+0000 7f8279686640 10 client.4445.objecter successfully canceled tick 2024-03-29T11:24:58.584 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.601+0000 7f8279686640 1 -- 172.21.15.104:0/1624767182 >> v1:172.21.15.104:6816/3341126660 conn(0x7f826006dd00 legacy=0x7f82600701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:58.584 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.601+0000 7f8279686640 10 monclient: shutdown 2024-03-29T11:24:58.584 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.601+0000 7f8279686640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:58.584 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.601+0000 7f8279686640 1 -- 172.21.15.104:0/1624767182 >> v1:172.21.15.104:6790/0 conn(0x7f827414a700 legacy=0x7f82741e7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:58.584 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.601+0000 7f8279686640 1 -- 172.21.15.104:0/1624767182 shutdown_connections 2024-03-29T11:24:58.584 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.601+0000 7f8279686640 1 -- 172.21.15.104:0/1624767182 >> 172.21.15.104:0/1624767182 conn(0x7f827406b660 msgr2=0x7f827406ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:58.584 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.601+0000 7f8279686640 1 -- 172.21.15.104:0/1624767182 shutdown_connections 2024-03-29T11:24:58.584 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.601+0000 7f8279686640 1 -- 172.21.15.104:0/1624767182 wait complete. 2024-03-29T11:24:58.584 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:24:58.601+0000 7f8279686640 1 librados: shutdown 2024-03-29T11:24:58.605 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:05.412 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:05.554 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.571+0000 7f3d739a3640 10 monclient: get_monmap_and_config 2024-03-29T11:25:05.554 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.571+0000 7f3d739a3640 10 monclient: build_initial_monmap 2024-03-29T11:25:05.555 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.571+0000 7f3d739a3640 10 monclient: monmap: 2024-03-29T11:25:05.555 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:25:05.555 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:25:05.555 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:25:05.572232+0000 2024-03-29T11:25:05.555 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:25:05.572232+0000 2024-03-29T11:25:05.555 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:05.555 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:25:05.555 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:25:05.555 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:25:05.555 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:25:05.555 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:25:05.556 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.572+0000 7f3d739a3640 1 Processor -- start 2024-03-29T11:25:05.556 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.573+0000 7f3d739a3640 1 -- start start 2024-03-29T11:25:05.556 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.573+0000 7f3d739a3640 10 monclient: init 2024-03-29T11:25:05.558 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.574+0000 7f3d739a3640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:05.558 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.574+0000 7f3d739a3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:25:05.558 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.574+0000 7f3d739a3640 10 monclient(hunting): picked mon.noname-a con 0x7f3d6c0b0770 addr v1:172.21.15.104:6789/0 2024-03-29T11:25:05.558 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.574+0000 7f3d739a3640 10 monclient(hunting): picked mon.noname-c con 0x7f3d6c14a700 addr v1:172.21.15.119:6789/0 2024-03-29T11:25:05.558 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.574+0000 7f3d739a3640 10 monclient(hunting): picked mon.noname-b con 0x7f3d6c0ac5a0 addr v1:172.21.15.104:6790/0 2024-03-29T11:25:05.558 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.574+0000 7f3d739a3640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d6c0aba70 con 0x7f3d6c0b0770 2024-03-29T11:25:05.558 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.574+0000 7f3d739a3640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d6c0aecb0 con 0x7f3d6c14a700 2024-03-29T11:25:05.558 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.574+0000 7f3d739a3640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d6c0af150 con 0x7f3d6c0ac5a0 2024-03-29T11:25:05.558 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.574+0000 7f3d739a3640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:05.558 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.574+0000 7f3d739a3640 10 monclient(hunting): authenticate will time out at 1286.825227s 2024-03-29T11:25:05.559 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.576+0000 7f3d70f3e640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f3d6c14a700 0x7f3d6c0abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:45362/0 (socket says 172.21.15.104:45362) 2024-03-29T11:25:05.559 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.576+0000 7f3d70f3e640 1 -- 172.21.15.104:0/2498290869 learned_addr learned my addr 172.21.15.104:0/2498290869 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:25:05.560 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.577+0000 7f3d63fff640 1 -- 172.21.15.104:0/2498290869 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 475970134 0 0) 0x7f3d6c0aecb0 con 0x7f3d6c14a700 2024-03-29T11:25:05.561 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.577+0000 7f3d63fff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:05.561 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.577+0000 7f3d63fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:05.561 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.577+0000 7f3d63fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:05.561 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.577+0000 7f3d63fff640 10 monclient(hunting): my global_id is 4417 2024-03-29T11:25:05.561 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.577+0000 7f3d63fff640 1 -- 172.21.15.104:0/2498290869 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d58003520 con 0x7f3d6c14a700 2024-03-29T11:25:05.561 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.577+0000 7f3d63fff640 1 -- 172.21.15.104:0/2498290869 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2058931311 0 0) 0x7f3d58003520 con 0x7f3d6c14a700 2024-03-29T11:25:05.561 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.577+0000 7f3d63fff640 1 -- 172.21.15.104:0/2498290869 >> v1:172.21.15.104:6790/0 conn(0x7f3d6c0ac5a0 legacy=0x7f3d6c151f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:05.562 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.578+0000 7f3d63fff640 1 -- 172.21.15.104:0/2498290869 >> v1:172.21.15.104:6789/0 conn(0x7f3d6c0b0770 legacy=0x7f3d6c149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:05.562 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.578+0000 7f3d63fff640 10 monclient: _finish_hunting 0 2024-03-29T11:25:05.562 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.578+0000 7f3d63fff640 1 monclient: found mon.noname-c 2024-03-29T11:25:05.562 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.578+0000 7f3d63fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:05.562 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.578+0000 7f3d63fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.119:6789/0 2024-03-29T11:25:05.563 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.578+0000 7f3d63fff640 1 -- 172.21.15.104:0/2498290869 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d6c0ac9d0 con 0x7f3d6c14a700 2024-03-29T11:25:05.563 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.578+0000 7f3d63fff640 10 monclient: _finish_auth 0 2024-03-29T11:25:05.563 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.578+0000 7f3d63fff640 10 monclient: _check_auth_tickets 2024-03-29T11:25:05.563 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.578+0000 7f3d63fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:05.563 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.578+0000 7f3d63fff640 1 -- 172.21.15.104:0/2498290869 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f3d68001670 con 0x7f3d6c14a700 2024-03-29T11:25:05.563 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.578+0000 7f3d63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:05.563 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.578+0000 7f3d63fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:05.563 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.578+0000 7f3d63fff640 10 monclient: dump: 2024-03-29T11:25:05.563 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:25:05.563 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:25:05.563 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:05.563 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:05.563 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:05.563 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:25:05.564 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:25:05.564 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:25:05.564 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:25:05.564 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:25:05.564 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.579+0000 7f3d739a3640 5 monclient: authenticate success, global_id 4417 2024-03-29T11:25:05.564 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.579+0000 7f3d739a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:05.565 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.580+0000 7f3d63fff640 1 -- 172.21.15.104:0/2498290869 <== mon.1 v1:172.21.15.119:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d68001900 con 0x7f3d6c14a700 2024-03-29T11:25:05.565 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.580+0000 7f3d63fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:05.565 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.580+0000 7f3d739a3640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:05.565 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.580+0000 7f3d739a3640 4 set_mon_vals no callback set 2024-03-29T11:25:05.565 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.580+0000 7f3d739a3640 10 monclient: shutdown 2024-03-29T11:25:05.565 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.580+0000 7f3d739a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:05.565 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.580+0000 7f3d739a3640 1 -- 172.21.15.104:0/2498290869 >> v1:172.21.15.119:6789/0 conn(0x7f3d6c14a700 legacy=0x7f3d6c0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:05.565 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.580+0000 7f3d63fff640 1 -- 172.21.15.104:0/2498290869 <== mon.1 v1:172.21.15.119:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f3d68001cd0 con 0x7f3d6c14a700 2024-03-29T11:25:05.565 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.580+0000 7f3d63fff640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:25:05.565 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.581+0000 7f3d739a3640 1 -- 172.21.15.104:0/2498290869 shutdown_connections 2024-03-29T11:25:05.565 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.581+0000 7f3d739a3640 1 -- 172.21.15.104:0/2498290869 >> 172.21.15.104:0/2498290869 conn(0x7f3d6c06b660 msgr2=0x7f3d6c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:05.566 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.581+0000 7f3d739a3640 1 -- 172.21.15.104:0/2498290869 shutdown_connections 2024-03-29T11:25:05.566 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.581+0000 7f3d739a3640 1 -- 172.21.15.104:0/2498290869 wait complete. 2024-03-29T11:25:05.566 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.582+0000 7f3d739a3640 10 monclient: build_initial_monmap 2024-03-29T11:25:05.566 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.582+0000 7f3d739a3640 10 monclient: monmap: 2024-03-29T11:25:05.567 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:25:05.567 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:05.567 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:25:05.567 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:25:05.567 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:05.567 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:25:05.567 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:25:05.567 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:25:05.567 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:25:05.567 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:25:05.567 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.583+0000 7f3d739a3640 1 librados: starting msgr at 2024-03-29T11:25:05.567 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.583+0000 7f3d739a3640 1 librados: starting objecter 2024-03-29T11:25:05.567 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.583+0000 7f3d739a3640 1 Processor -- start 2024-03-29T11:25:05.569 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.584+0000 7f3d739a3640 1 -- start start 2024-03-29T11:25:05.569 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.584+0000 7f3d739a3640 1 librados: setting wanted keys 2024-03-29T11:25:05.569 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.584+0000 7f3d739a3640 1 librados: calling monclient init 2024-03-29T11:25:05.569 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.584+0000 7f3d739a3640 10 monclient: init 2024-03-29T11:25:05.569 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.584+0000 7f3d739a3640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:05.570 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.584+0000 7f3d739a3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:25:05.570 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.585+0000 7f3d739a3640 10 monclient(hunting): picked mon.noname-c con 0x7f3d6c0ac5a0 addr v1:172.21.15.104:6790/0 2024-03-29T11:25:05.570 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.585+0000 7f3d739a3640 10 monclient(hunting): picked mon.noname-b con 0x7f3d6c0b0770 addr v1:172.21.15.119:6789/0 2024-03-29T11:25:05.570 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.585+0000 7f3d739a3640 10 monclient(hunting): picked mon.noname-a con 0x7f3d6c14a700 addr v1:172.21.15.104:6789/0 2024-03-29T11:25:05.570 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.585+0000 7f3d739a3640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d6c1282a0 con 0x7f3d6c14a700 2024-03-29T11:25:05.570 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.585+0000 7f3d739a3640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d6c1459e0 con 0x7f3d6c0b0770 2024-03-29T11:25:05.570 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.585+0000 7f3d739a3640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d6c127180 con 0x7f3d6c0ac5a0 2024-03-29T11:25:05.570 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.585+0000 7f3d739a3640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:05.570 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.585+0000 7f3d739a3640 10 monclient(hunting): authenticate will time out at 1286.836390s 2024-03-29T11:25:05.575 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.587+0000 7f3d70f3e640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f3d6c0b0770 0x7f3d6c1db7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:46022/0 (socket says 172.21.15.104:46022) 2024-03-29T11:25:05.575 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.587+0000 7f3d70f3e640 1 -- 172.21.15.104:0/3593974344 learned_addr learned my addr 172.21.15.104:0/3593974344 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:25:05.575 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.587+0000 7f3d61ffb640 10 client.?.objecter ms_handle_connect 0x7f3d6c0b0770 2024-03-29T11:25:05.575 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.587+0000 7f3d61ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:05.576 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.587+0000 7f3d61ffb640 1 -- 172.21.15.104:0/3593974344 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 587590186 0 0) 0x7f3d6c1459e0 con 0x7f3d6c0b0770 2024-03-29T11:25:05.576 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.587+0000 7f3d61ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:05.576 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.587+0000 7f3d61ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:05.576 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.587+0000 7f3d61ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:05.576 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.587+0000 7f3d61ffb640 10 monclient(hunting): my global_id is 4420 2024-03-29T11:25:05.576 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.587+0000 7f3d61ffb640 1 -- 172.21.15.104:0/3593974344 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d50004eb0 con 0x7f3d6c0b0770 2024-03-29T11:25:05.576 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.588+0000 7f3d61ffb640 10 client.?.objecter ms_handle_connect 0x7f3d6c14a700 2024-03-29T11:25:05.576 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.588+0000 7f3d61ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:05.576 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.589+0000 7f3d61ffb640 10 client.?.objecter ms_handle_connect 0x7f3d6c0ac5a0 2024-03-29T11:25:05.576 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.589+0000 7f3d61ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:05.576 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.589+0000 7f3d61ffb640 1 -- 172.21.15.104:0/3593974344 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3495352559 0 0) 0x7f3d6c127180 con 0x7f3d6c0ac5a0 2024-03-29T11:25:05.576 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.589+0000 7f3d61ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:05.576 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.589+0000 7f3d61ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:05.577 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.589+0000 7f3d61ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:05.577 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.589+0000 7f3d61ffb640 10 monclient(hunting): my global_id is 4448 2024-03-29T11:25:05.577 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.589+0000 7f3d61ffb640 1 -- 172.21.15.104:0/3593974344 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d6c1459e0 con 0x7f3d6c0ac5a0 2024-03-29T11:25:05.577 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.589+0000 7f3d61ffb640 1 -- 172.21.15.104:0/3593974344 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2831820093 0 0) 0x7f3d50004eb0 con 0x7f3d6c0b0770 2024-03-29T11:25:05.577 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.589+0000 7f3d61ffb640 1 -- 172.21.15.104:0/3593974344 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d6c127180 con 0x7f3d6c0b0770 2024-03-29T11:25:05.577 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.589+0000 7f3d61ffb640 1 -- 172.21.15.104:0/3593974344 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1034371334 0 0) 0x7f3d6c1282a0 con 0x7f3d6c14a700 2024-03-29T11:25:05.577 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.589+0000 7f3d61ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:05.577 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.589+0000 7f3d61ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:05.577 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.589+0000 7f3d61ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:05.577 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.589+0000 7f3d61ffb640 10 monclient(hunting): my global_id is 4449 2024-03-29T11:25:05.577 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.589+0000 7f3d61ffb640 1 -- 172.21.15.104:0/3593974344 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d50004eb0 con 0x7f3d6c14a700 2024-03-29T11:25:05.577 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.589+0000 7f3d61ffb640 1 -- 172.21.15.104:0/3593974344 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f3d68001630 con 0x7f3d6c0b0770 2024-03-29T11:25:05.577 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.589+0000 7f3d61ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:05.577 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.589+0000 7f3d61ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:05.578 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.589+0000 7f3d61ffb640 10 monclient(hunting): dump: 2024-03-29T11:25:05.578 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:25:05.578 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:25:05.578 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:05.578 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:05.578 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:05.578 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:25:05.578 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:25:05.578 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:25:05.578 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:25:05.578 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:25:05.578 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.589+0000 7f3d61ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:05.582 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.590+0000 7f3d61ffb640 1 -- 172.21.15.104:0/3593974344 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 122709125 0 0) 0x7f3d6c1459e0 con 0x7f3d6c0ac5a0 2024-03-29T11:25:05.582 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.590+0000 7f3d61ffb640 1 -- 172.21.15.104:0/3593974344 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d6c1282a0 con 0x7f3d6c0ac5a0 2024-03-29T11:25:05.582 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.590+0000 7f3d61ffb640 1 -- 172.21.15.104:0/3593974344 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f3d5c001830 con 0x7f3d6c0ac5a0 2024-03-29T11:25:05.582 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.590+0000 7f3d61ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:05.582 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.590+0000 7f3d61ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:05.582 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.590+0000 7f3d61ffb640 10 monclient(hunting): dump: 2024-03-29T11:25:05.582 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:25:05.582 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:25:05.582 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:05.582 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:05.582 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:05.582 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:25:05.582 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:25:05.583 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:25:05.583 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:25:05.583 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:25:05.583 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.590+0000 7f3d61ffb640 1 -- 172.21.15.104:0/3593974344 <== mon.2 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2249365644 0 0) 0x7f3d6c1282a0 con 0x7f3d6c0ac5a0 2024-03-29T11:25:05.583 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.590+0000 7f3d61ffb640 1 -- 172.21.15.104:0/3593974344 >> v1:172.21.15.119:6789/0 conn(0x7f3d6c0b0770 legacy=0x7f3d6c1db7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:05.583 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.590+0000 7f3d61ffb640 1 -- 172.21.15.104:0/3593974344 >> v1:172.21.15.104:6789/0 conn(0x7f3d6c14a700 legacy=0x7f3d6c1dbec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:05.583 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.590+0000 7f3d61ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:25:05.583 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.590+0000 7f3d61ffb640 1 monclient: found mon.c 2024-03-29T11:25:05.583 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.590+0000 7f3d61ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:05.583 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.590+0000 7f3d61ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:25:05.583 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.590+0000 7f3d61ffb640 1 -- 172.21.15.104:0/3593974344 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d6c126c10 con 0x7f3d6c0ac5a0 2024-03-29T11:25:05.583 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.590+0000 7f3d61ffb640 10 monclient: _finish_auth 0 2024-03-29T11:25:05.583 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.590+0000 7f3d61ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:25:05.585 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.590+0000 7f3d61ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:05.585 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.591+0000 7f3d739a3640 5 monclient: authenticate success, global_id 4448 2024-03-29T11:25:05.585 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.591+0000 7f3d739a3640 10 monclient: _renew_subs 2024-03-29T11:25:05.585 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.591+0000 7f3d739a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:25:05.585 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.591+0000 7f3d739a3640 1 -- 172.21.15.104:0/3593974344 --> v1:172.21.15.104:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d6c126e60 con 0x7f3d6c0ac5a0 2024-03-29T11:25:05.585 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.591+0000 7f3d739a3640 10 client.4448.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:05.585 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.591+0000 7f3d739a3640 10 monclient: _renew_subs 2024-03-29T11:25:05.585 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.591+0000 7f3d739a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:25:05.585 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.591+0000 7f3d739a3640 1 -- 172.21.15.104:0/3593974344 --> v1:172.21.15.104:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d6c1e71e0 con 0x7f3d6c0ac5a0 2024-03-29T11:25:05.585 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.591+0000 7f3d739a3640 1 librados: init done 2024-03-29T11:25:05.585 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.594+0000 7f3d577fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:05.586 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.594+0000 7f3d61ffb640 1 -- 172.21.15.104:0/3593974344 <== mon.2 v1:172.21.15.104:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d5c001990 con 0x7f3d6c0ac5a0 2024-03-29T11:25:05.586 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.594+0000 7f3d577fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:05.586 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.594+0000 7f3d577fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:25:05.586 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.594+0000 7f3d577fe640 1 -- 172.21.15.104:0/3593974344 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d4c005080 con 0x7f3d6c0ac5a0 2024-03-29T11:25:05.586 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.594+0000 7f3d61ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:05.586 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.594+0000 7f3d61ffb640 1 -- 172.21.15.104:0/3593974344 <== mon.2 v1:172.21.15.104:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f3d5c0037b0 con 0x7f3d6c0ac5a0 2024-03-29T11:25:05.586 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.594+0000 7f3d61ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:05.586 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.594+0000 7f3d61ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:05.586 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.594+0000 7f3d61ffb640 10 monclient: dump: 2024-03-29T11:25:05.586 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:25:05.586 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:25:05.586 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:05.586 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:05.587 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:05.587 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:25:05.587 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:25:05.587 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:25:05.587 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:25:05.587 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:25:05.587 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.594+0000 7f3d61ffb640 1 -- 172.21.15.104:0/3593974344 <== mon.2 v1:172.21.15.104:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f3d5c0039d0 con 0x7f3d6c0ac5a0 2024-03-29T11:25:05.587 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.594+0000 7f3d62ffd640 4 set_mon_vals no callback set 2024-03-29T11:25:05.587 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.595+0000 7f3d61ffb640 1 -- 172.21.15.104:0/3593974344 <== mon.2 v1:172.21.15.104:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f3d5c083880 con 0x7f3d6c0ac5a0 2024-03-29T11:25:05.587 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.595+0000 7f3d61ffb640 10 client.4448.objecter ms_dispatch 0x7f3d6c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:05.587 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.595+0000 7f3d61ffb640 3 client.4448.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:05.587 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.595+0000 7f3d61ffb640 3 client.4448.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:05.587 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.595+0000 7f3d61ffb640 20 client.4448.objecter dump_active .. 0 homeless 2024-03-29T11:25:05.587 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.596+0000 7f3d70f3e640 10 monclient: get_auth_request con 0x7f3d5006de80 auth_method 0 2024-03-29T11:25:05.588 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.597+0000 7f3d61ffb640 10 client.4448.objecter ms_handle_connect 0x7f3d5006de80 2024-03-29T11:25:05.588 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.603+0000 7f3d61ffb640 1 -- 172.21.15.104:0/3593974344 <== mon.2 v1:172.21.15.104:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3d5c04f280 con 0x7f3d6c0ac5a0 2024-03-29T11:25:05.588 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.603+0000 7f3d61ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:05.588 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.603+0000 7f3d61ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:05.908 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.925+0000 7f3d577fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:05.908 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.925+0000 7f3d577fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:05.909 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.925+0000 7f3d577fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:25:05.909 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.925+0000 7f3d577fe640 1 -- 172.21.15.104:0/3593974344 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d4c005330 con 0x7f3d6c0ac5a0 2024-03-29T11:25:05.909 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.926+0000 7f3d61ffb640 1 -- 172.21.15.104:0/3593974344 <== mon.2 v1:172.21.15.104:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f3d5c0511b0 con 0x7f3d6c0ac5a0 2024-03-29T11:25:05.909 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.926+0000 7f3d61ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:05.909 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.926+0000 7f3d61ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:05.910 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:25:05.910 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:05.913 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.930+0000 7f3d577fe640 10 librados: watch_flush enter 2024-03-29T11:25:05.913 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.930+0000 7f3d577fe640 10 librados: watch_flush exit 2024-03-29T11:25:05.913 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.930+0000 7f3d577fe640 20 client.4448.objecter shutdown clearing up homeless session... 2024-03-29T11:25:05.913 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.930+0000 7f3d577fe640 10 client.4448.objecter successfully canceled tick 2024-03-29T11:25:05.914 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.930+0000 7f3d577fe640 1 -- 172.21.15.104:0/3593974344 >> v1:172.21.15.104:6816/3341126660 conn(0x7f3d5006de80 legacy=0x7f3d50070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:05.915 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.931+0000 7f3d577fe640 10 monclient: shutdown 2024-03-29T11:25:05.915 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.931+0000 7f3d577fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:05.915 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.931+0000 7f3d577fe640 1 -- 172.21.15.104:0/3593974344 >> v1:172.21.15.104:6790/0 conn(0x7f3d6c0ac5a0 legacy=0x7f3d6c125e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:05.915 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.931+0000 7f3d577fe640 1 -- 172.21.15.104:0/3593974344 shutdown_connections 2024-03-29T11:25:05.915 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.931+0000 7f3d577fe640 1 -- 172.21.15.104:0/3593974344 >> 172.21.15.104:0/3593974344 conn(0x7f3d6c06b660 msgr2=0x7f3d6c0abe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:05.915 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.931+0000 7f3d577fe640 1 -- 172.21.15.104:0/3593974344 shutdown_connections 2024-03-29T11:25:05.915 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.931+0000 7f3d577fe640 1 -- 172.21.15.104:0/3593974344 wait complete. 2024-03-29T11:25:05.915 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:05.931+0000 7f3d577fe640 1 librados: shutdown 2024-03-29T11:25:05.931 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:12.742 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:12.898 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.915+0000 7f1dcbfff640 10 monclient: get_monmap_and_config 2024-03-29T11:25:12.899 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.915+0000 7f1dcbfff640 10 monclient: build_initial_monmap 2024-03-29T11:25:12.899 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.915+0000 7f1dcbfff640 10 monclient: monmap: 2024-03-29T11:25:12.899 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:25:12.899 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:25:12.899 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:25:12.916268+0000 2024-03-29T11:25:12.899 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:25:12.916268+0000 2024-03-29T11:25:12.899 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:12.899 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:25:12.899 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:25:12.899 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:25:12.899 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:25:12.899 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:25:12.900 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.916+0000 7f1dcbfff640 1 Processor -- start 2024-03-29T11:25:12.900 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.916+0000 7f1dcbfff640 1 -- start start 2024-03-29T11:25:12.900 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.916+0000 7f1dcbfff640 10 monclient: init 2024-03-29T11:25:12.900 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.917+0000 7f1dcbfff640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:12.900 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.917+0000 7f1dcbfff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:25:12.900 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.917+0000 7f1dcbfff640 10 monclient(hunting): picked mon.noname-c con 0x7f1dcc06fb00 addr v1:172.21.15.119:6789/0 2024-03-29T11:25:12.901 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.917+0000 7f1dcbfff640 10 monclient(hunting): picked mon.noname-b con 0x7f1dcc0b0420 addr v1:172.21.15.104:6790/0 2024-03-29T11:25:12.901 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.917+0000 7f1dcbfff640 10 monclient(hunting): picked mon.noname-a con 0x7f1dcc0af990 addr v1:172.21.15.104:6789/0 2024-03-29T11:25:12.901 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.917+0000 7f1dcbfff640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dcc140c10 con 0x7f1dcc0af990 2024-03-29T11:25:12.901 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.917+0000 7f1dcbfff640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dcc071fa0 con 0x7f1dcc06fb00 2024-03-29T11:25:12.901 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.917+0000 7f1dcbfff640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dcc072280 con 0x7f1dcc0b0420 2024-03-29T11:25:12.901 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.917+0000 7f1dcbfff640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:12.901 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.917+0000 7f1dcbfff640 10 monclient(hunting): authenticate will time out at 1294.168293s 2024-03-29T11:25:12.904 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.920+0000 7f1dcb7fe640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f1dcc0af990 0x7f1dcc14b6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:41545/0 (socket says 172.21.15.104:41545) 2024-03-29T11:25:12.904 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.920+0000 7f1dcb7fe640 1 -- 172.21.15.104:0/1815781940 learned_addr learned my addr 172.21.15.104:0/1815781940 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:25:12.905 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.921+0000 7f1dc9ffb640 1 -- 172.21.15.104:0/1815781940 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1124478715 0 0) 0x7f1dcc140c10 con 0x7f1dcc0af990 2024-03-29T11:25:12.905 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.921+0000 7f1dc9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:12.905 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.921+0000 7f1dc9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:12.905 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.921+0000 7f1dc9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:12.905 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.921+0000 7f1dc9ffb640 10 monclient(hunting): my global_id is 4452 2024-03-29T11:25:12.905 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.921+0000 7f1dc9ffb640 1 -- 172.21.15.104:0/1815781940 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1db4003520 con 0x7f1dcc0af990 2024-03-29T11:25:12.905 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.921+0000 7f1dc9ffb640 1 -- 172.21.15.104:0/1815781940 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2557888305 0 0) 0x7f1dcc072280 con 0x7f1dcc0b0420 2024-03-29T11:25:12.905 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.921+0000 7f1dc9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:12.905 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.921+0000 7f1dc9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:12.905 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.921+0000 7f1dc9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:12.905 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.921+0000 7f1dc9ffb640 10 monclient(hunting): my global_id is 4451 2024-03-29T11:25:12.905 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.921+0000 7f1dc9ffb640 1 -- 172.21.15.104:0/1815781940 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dcc140c10 con 0x7f1dcc0b0420 2024-03-29T11:25:12.906 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.921+0000 7f1dc9ffb640 1 -- 172.21.15.104:0/1815781940 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 224666418 0 0) 0x7f1db4003520 con 0x7f1dcc0af990 2024-03-29T11:25:12.906 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.921+0000 7f1dc9ffb640 1 -- 172.21.15.104:0/1815781940 >> v1:172.21.15.104:6790/0 conn(0x7f1dcc0b0420 legacy=0x7f1dcc140500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:12.906 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.921+0000 7f1dc9ffb640 1 -- 172.21.15.104:0/1815781940 >> v1:172.21.15.119:6789/0 conn(0x7f1dcc06fb00 legacy=0x7f1dcc06fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:12.906 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.921+0000 7f1dc9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:25:12.906 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.921+0000 7f1dc9ffb640 1 monclient: found mon.noname-a 2024-03-29T11:25:12.906 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.921+0000 7f1dc9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:12.906 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.921+0000 7f1dc9ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.104:6789/0 2024-03-29T11:25:12.906 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.921+0000 7f1dc9ffb640 1 -- 172.21.15.104:0/1815781940 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dcc0710a0 con 0x7f1dcc0af990 2024-03-29T11:25:12.906 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.921+0000 7f1dc9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:25:12.906 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.921+0000 7f1dc9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:25:12.906 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.921+0000 7f1dc9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:12.906 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.921+0000 7f1dcbfff640 5 monclient: authenticate success, global_id 4452 2024-03-29T11:25:12.906 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.921+0000 7f1dcbfff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:12.906 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.921+0000 7f1dc9ffb640 1 -- 172.21.15.104:0/1815781940 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f1dc4001a10 con 0x7f1dcc0af990 2024-03-29T11:25:12.907 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.921+0000 7f1dc9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:12.907 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.921+0000 7f1dc9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:12.907 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.921+0000 7f1dc9ffb640 10 monclient: dump: 2024-03-29T11:25:12.907 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:25:12.907 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:25:12.907 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:12.907 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:12.907 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:12.907 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:25:12.907 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:25:12.908 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:25:12.908 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:25:12.908 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:25:12.908 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.921+0000 7f1dcbfff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:12.908 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.922+0000 7f1dc9ffb640 1 -- 172.21.15.104:0/1815781940 <== mon.0 v1:172.21.15.104:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1dc4001c60 con 0x7f1dcc0af990 2024-03-29T11:25:12.909 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.922+0000 7f1dc9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:12.909 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.922+0000 7f1dcbfff640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:12.909 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.922+0000 7f1dcbfff640 4 set_mon_vals no callback set 2024-03-29T11:25:12.909 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.923+0000 7f1dcbfff640 10 monclient: shutdown 2024-03-29T11:25:12.909 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.923+0000 7f1dcbfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:12.909 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.923+0000 7f1dcbfff640 1 -- 172.21.15.104:0/1815781940 >> v1:172.21.15.104:6789/0 conn(0x7f1dcc0af990 legacy=0x7f1dcc14b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:12.909 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.923+0000 7f1dc9ffb640 1 -- 172.21.15.104:0/1815781940 <== mon.0 v1:172.21.15.104:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f1dc4001ff0 con 0x7f1dcc0af990 2024-03-29T11:25:12.909 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.923+0000 7f1dc9ffb640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:25:12.909 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.924+0000 7f1dcbfff640 1 -- 172.21.15.104:0/1815781940 shutdown_connections 2024-03-29T11:25:12.909 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.924+0000 7f1dcbfff640 1 -- 172.21.15.104:0/1815781940 >> 172.21.15.104:0/1815781940 conn(0x7f1dcc0672e0 msgr2=0x7f1dcc0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:12.909 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.925+0000 7f1dcbfff640 1 -- 172.21.15.104:0/1815781940 shutdown_connections 2024-03-29T11:25:12.909 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.925+0000 7f1dcbfff640 1 -- 172.21.15.104:0/1815781940 wait complete. 2024-03-29T11:25:12.910 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.926+0000 7f1dcbfff640 10 monclient: build_initial_monmap 2024-03-29T11:25:12.910 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.926+0000 7f1dcbfff640 10 monclient: monmap: 2024-03-29T11:25:12.910 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:25:12.910 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:12.911 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:25:12.911 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:25:12.911 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:12.911 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:25:12.911 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:25:12.911 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:25:12.911 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:25:12.911 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:25:12.911 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.926+0000 7f1dcbfff640 1 librados: starting msgr at 2024-03-29T11:25:12.912 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.926+0000 7f1dcbfff640 1 librados: starting objecter 2024-03-29T11:25:12.912 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.927+0000 7f1dcbfff640 1 Processor -- start 2024-03-29T11:25:12.912 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.927+0000 7f1dcbfff640 1 -- start start 2024-03-29T11:25:12.912 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.927+0000 7f1dcbfff640 1 librados: setting wanted keys 2024-03-29T11:25:12.912 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.927+0000 7f1dcbfff640 1 librados: calling monclient init 2024-03-29T11:25:12.912 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.927+0000 7f1dcbfff640 10 monclient: init 2024-03-29T11:25:12.913 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.928+0000 7f1dcbfff640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:12.913 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.928+0000 7f1dcbfff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:25:12.913 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.928+0000 7f1dcbfff640 10 monclient(hunting): picked mon.noname-a con 0x7f1dcc06fb00 addr v1:172.21.15.104:6789/0 2024-03-29T11:25:12.913 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.928+0000 7f1dcbfff640 10 monclient(hunting): picked mon.noname-c con 0x7f1dcc0af990 addr v1:172.21.15.104:6790/0 2024-03-29T11:25:12.913 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.928+0000 7f1dcbfff640 10 monclient(hunting): picked mon.noname-b con 0x7f1dcc0b0420 addr v1:172.21.15.119:6789/0 2024-03-29T11:25:12.914 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.928+0000 7f1dcbfff640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dcc138260 con 0x7f1dcc06fb00 2024-03-29T11:25:12.914 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.928+0000 7f1dcbfff640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dcc139470 con 0x7f1dcc0b0420 2024-03-29T11:25:12.914 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.928+0000 7f1dcbfff640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dcc1390e0 con 0x7f1dcc0af990 2024-03-29T11:25:12.914 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.928+0000 7f1dcbfff640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:12.914 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.928+0000 7f1dcbfff640 10 monclient(hunting): authenticate will time out at 1294.179225s 2024-03-29T11:25:12.914 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.928+0000 7f1dcb7fe640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f1dcc0b0420 0x7f1dcc1d8370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:52756/0 (socket says 172.21.15.104:52756) 2024-03-29T11:25:12.914 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.928+0000 7f1dcb7fe640 1 -- 172.21.15.104:0/2692019514 learned_addr learned my addr 172.21.15.104:0/2692019514 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:25:12.914 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.928+0000 7f1db3fff640 10 client.?.objecter ms_handle_connect 0x7f1dcc0b0420 2024-03-29T11:25:12.914 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.928+0000 7f1db3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:12.914 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.929+0000 7f1db3fff640 1 -- 172.21.15.104:0/2692019514 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 493741321 0 0) 0x7f1dcc139470 con 0x7f1dcc0b0420 2024-03-29T11:25:12.915 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.929+0000 7f1db3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:12.915 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.929+0000 7f1db3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:12.915 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.929+0000 7f1db3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:12.915 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.929+0000 7f1db3fff640 10 monclient(hunting): my global_id is 4423 2024-03-29T11:25:12.915 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.929+0000 7f1db3fff640 1 -- 172.21.15.104:0/2692019514 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1db8004eb0 con 0x7f1dcc0b0420 2024-03-29T11:25:12.915 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.929+0000 7f1db3fff640 1 -- 172.21.15.104:0/2692019514 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3294586384 0 0) 0x7f1db8004eb0 con 0x7f1dcc0b0420 2024-03-29T11:25:12.915 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.929+0000 7f1db3fff640 1 -- 172.21.15.104:0/2692019514 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1dcc139470 con 0x7f1dcc0b0420 2024-03-29T11:25:12.915 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.930+0000 7f1db3fff640 1 -- 172.21.15.104:0/2692019514 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f1dc4001f70 con 0x7f1dcc0b0420 2024-03-29T11:25:12.915 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.930+0000 7f1db3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:12.915 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.930+0000 7f1db3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:12.915 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.930+0000 7f1db3fff640 10 monclient(hunting): dump: 2024-03-29T11:25:12.915 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:25:12.915 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:25:12.915 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:12.915 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:12.916 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:12.916 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:25:12.916 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:25:12.916 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:25:12.916 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:25:12.916 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:25:12.916 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.930+0000 7f1db3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:12.916 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.930+0000 7f1db3fff640 1 -- 172.21.15.104:0/2692019514 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1389228520 0 0) 0x7f1dcc139470 con 0x7f1dcc0b0420 2024-03-29T11:25:12.916 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.930+0000 7f1db3fff640 1 -- 172.21.15.104:0/2692019514 >> v1:172.21.15.104:6790/0 conn(0x7f1dcc0af990 legacy=0x7f1dcc131a90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:25:12.917 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.931+0000 7f1db3fff640 1 -- 172.21.15.104:0/2692019514 >> v1:172.21.15.104:6789/0 conn(0x7f1dcc06fb00 legacy=0x7f1dcc1389d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:12.917 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.931+0000 7f1db3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:25:12.917 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.931+0000 7f1db3fff640 1 monclient: found mon.b 2024-03-29T11:25:12.917 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.931+0000 7f1db3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:12.917 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.931+0000 7f1db3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:25:12.917 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.931+0000 7f1db3fff640 1 -- 172.21.15.104:0/2692019514 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dcc06dca0 con 0x7f1dcc0b0420 2024-03-29T11:25:12.917 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.931+0000 7f1db3fff640 10 monclient: _finish_auth 0 2024-03-29T11:25:12.917 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.931+0000 7f1db3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:25:12.917 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.931+0000 7f1db3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:12.917 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.931+0000 7f1dcbfff640 5 monclient: authenticate success, global_id 4423 2024-03-29T11:25:12.917 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.931+0000 7f1dcbfff640 10 monclient: _renew_subs 2024-03-29T11:25:12.918 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.931+0000 7f1dcbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:25:12.918 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.931+0000 7f1dcbfff640 1 -- 172.21.15.104:0/2692019514 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dcc13c0f0 con 0x7f1dcc0b0420 2024-03-29T11:25:12.918 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.931+0000 7f1dcbfff640 10 client.4423.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:12.918 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.931+0000 7f1dcbfff640 10 monclient: _renew_subs 2024-03-29T11:25:12.918 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.931+0000 7f1dcbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:25:12.918 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.931+0000 7f1dcbfff640 1 -- 172.21.15.104:0/2692019514 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dcc132380 con 0x7f1dcc0b0420 2024-03-29T11:25:12.918 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.931+0000 7f1dcbfff640 1 librados: init done 2024-03-29T11:25:12.918 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.931+0000 7f1db3fff640 1 -- 172.21.15.104:0/2692019514 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1dc4004f70 con 0x7f1dcc0b0420 2024-03-29T11:25:12.918 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.931+0000 7f1db3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:12.918 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.931+0000 7f1db3fff640 1 -- 172.21.15.104:0/2692019514 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f1dc4005380 con 0x7f1dcc0b0420 2024-03-29T11:25:12.918 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.931+0000 7f1db3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:12.918 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.931+0000 7f1db3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:12.918 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.931+0000 7f1db3fff640 10 monclient: dump: 2024-03-29T11:25:12.918 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:25:12.918 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:25:12.919 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:12.919 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:12.919 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:12.919 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:25:12.919 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:25:12.919 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:25:12.919 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:25:12.919 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:25:12.919 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.931+0000 7f1dc97fa640 4 set_mon_vals no callback set 2024-03-29T11:25:12.920 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.933+0000 7f1db3fff640 1 -- 172.21.15.104:0/2692019514 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f1dc40054e0 con 0x7f1dcc0b0420 2024-03-29T11:25:12.920 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.934+0000 7f1db3fff640 1 -- 172.21.15.104:0/2692019514 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f1dc4085050 con 0x7f1dcc0b0420 2024-03-29T11:25:12.920 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.934+0000 7f1db3fff640 10 client.4423.objecter ms_dispatch 0x7f1dcc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:12.920 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.934+0000 7f1db3fff640 3 client.4423.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:12.920 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.934+0000 7f1db3fff640 3 client.4423.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:12.921 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.934+0000 7f1db3fff640 20 client.4423.objecter dump_active .. 0 homeless 2024-03-29T11:25:12.921 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.934+0000 7f1dcaffd640 10 monclient: get_auth_request con 0x7f1db806db60 auth_method 0 2024-03-29T11:25:12.921 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.934+0000 7f1dcbfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:12.921 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.934+0000 7f1dcbfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:12.921 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.934+0000 7f1dcbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:25:12.921 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.934+0000 7f1dcbfff640 1 -- 172.21.15.104:0/2692019514 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dac005080 con 0x7f1dcc0b0420 2024-03-29T11:25:12.921 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.935+0000 7f1db3fff640 10 client.4423.objecter ms_handle_connect 0x7f1db806db60 2024-03-29T11:25:12.927 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.944+0000 7f1db3fff640 1 -- 172.21.15.104:0/2692019514 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1dc4050930 con 0x7f1dcc0b0420 2024-03-29T11:25:12.927 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.944+0000 7f1db3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:12.927 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:12.944+0000 7f1db3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:13.252 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:13.268+0000 7f1dcbfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:13.252 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:13.268+0000 7f1dcbfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:13.252 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:13.268+0000 7f1dcbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:25:13.252 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:13.268+0000 7f1dcbfff640 1 -- 172.21.15.104:0/2692019514 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1dac005330 con 0x7f1dcc0b0420 2024-03-29T11:25:13.252 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:13.269+0000 7f1db3fff640 1 -- 172.21.15.104:0/2692019514 <== mon.1 v1:172.21.15.119:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f1dc4052860 con 0x7f1dcc0b0420 2024-03-29T11:25:13.252 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:13.269+0000 7f1db3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:13.253 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:13.269+0000 7f1db3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:13.253 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:25:13.253 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:13.256 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:13.272+0000 7f1dcbfff640 10 librados: watch_flush enter 2024-03-29T11:25:13.256 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:13.272+0000 7f1dcbfff640 10 librados: watch_flush exit 2024-03-29T11:25:13.256 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:13.272+0000 7f1dcbfff640 20 client.4423.objecter shutdown clearing up homeless session... 2024-03-29T11:25:13.256 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:13.272+0000 7f1dcbfff640 10 client.4423.objecter successfully canceled tick 2024-03-29T11:25:13.257 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:13.273+0000 7f1dcbfff640 1 -- 172.21.15.104:0/2692019514 >> v1:172.21.15.104:6816/3341126660 conn(0x7f1db806db60 legacy=0x7f1db8070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:13.257 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:13.273+0000 7f1dcbfff640 10 monclient: shutdown 2024-03-29T11:25:13.257 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:13.273+0000 7f1dcbfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:13.257 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:13.273+0000 7f1dcbfff640 1 -- 172.21.15.104:0/2692019514 >> v1:172.21.15.119:6789/0 conn(0x7f1dcc0b0420 legacy=0x7f1dcc1d8370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:13.257 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:13.273+0000 7f1dcbfff640 1 -- 172.21.15.104:0/2692019514 shutdown_connections 2024-03-29T11:25:13.257 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:13.273+0000 7f1dcbfff640 1 -- 172.21.15.104:0/2692019514 >> 172.21.15.104:0/2692019514 conn(0x7f1dcc0672e0 msgr2=0x7f1dcc0708c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:13.258 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:13.274+0000 7f1dcbfff640 1 -- 172.21.15.104:0/2692019514 shutdown_connections 2024-03-29T11:25:13.258 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:13.274+0000 7f1dcbfff640 1 -- 172.21.15.104:0/2692019514 wait complete. 2024-03-29T11:25:13.258 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:13.275+0000 7f1dcbfff640 1 librados: shutdown 2024-03-29T11:25:13.275 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:20.082 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:20.235 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.249+0000 7fd6dbfff640 10 monclient: get_monmap_and_config 2024-03-29T11:25:20.235 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.249+0000 7fd6dbfff640 10 monclient: build_initial_monmap 2024-03-29T11:25:20.235 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.249+0000 7fd6dbfff640 10 monclient: monmap: 2024-03-29T11:25:20.235 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:25:20.235 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:25:20.235 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:25:20.251259+0000 2024-03-29T11:25:20.235 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:25:20.251259+0000 2024-03-29T11:25:20.235 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:20.235 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:25:20.235 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:25:20.235 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:25:20.236 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:25:20.236 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:25:20.237 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.253+0000 7fd6dbfff640 1 Processor -- start 2024-03-29T11:25:20.238 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.253+0000 7fd6dbfff640 1 -- start start 2024-03-29T11:25:20.238 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.253+0000 7fd6dbfff640 10 monclient: init 2024-03-29T11:25:20.241 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.254+0000 7fd6dbfff640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:20.241 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.254+0000 7fd6dbfff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:25:20.241 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.254+0000 7fd6dbfff640 10 monclient(hunting): picked mon.noname-c con 0x7fd6d4006410 addr v1:172.21.15.119:6789/0 2024-03-29T11:25:20.241 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.254+0000 7fd6dbfff640 10 monclient(hunting): picked mon.noname-b con 0x7fd6d402a640 addr v1:172.21.15.104:6790/0 2024-03-29T11:25:20.241 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.254+0000 7fd6dbfff640 10 monclient(hunting): picked mon.noname-a con 0x7fd6d40327d0 addr v1:172.21.15.104:6789/0 2024-03-29T11:25:20.241 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.254+0000 7fd6dbfff640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6d402c820 con 0x7fd6d40327d0 2024-03-29T11:25:20.241 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.254+0000 7fd6dbfff640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6d402a120 con 0x7fd6d4006410 2024-03-29T11:25:20.241 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.254+0000 7fd6dbfff640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6d40c75b0 con 0x7fd6d402a640 2024-03-29T11:25:20.241 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.254+0000 7fd6dbfff640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:20.241 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.254+0000 7fd6dbfff640 10 monclient(hunting): authenticate will time out at 1301.506637s 2024-03-29T11:25:20.247 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.261+0000 7fd6daffd640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fd6d4006410 0x7fd6d40c6ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:54549/0 (socket says 172.21.15.104:54549) 2024-03-29T11:25:20.248 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.261+0000 7fd6daffd640 1 -- 172.21.15.104:0/125746865 learned_addr learned my addr 172.21.15.104:0/125746865 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:25:20.248 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.261+0000 7fd6d9ffb640 1 -- 172.21.15.104:0/125746865 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1572742096 0 0) 0x7fd6d402a120 con 0x7fd6d4006410 2024-03-29T11:25:20.248 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.261+0000 7fd6d9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:20.248 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.261+0000 7fd6d9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:20.248 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.261+0000 7fd6d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:20.248 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.261+0000 7fd6d9ffb640 10 monclient(hunting): my global_id is 4426 2024-03-29T11:25:20.248 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.261+0000 7fd6d9ffb640 1 -- 172.21.15.104:0/125746865 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6c4003520 con 0x7fd6d4006410 2024-03-29T11:25:20.248 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.261+0000 7fd6d9ffb640 1 -- 172.21.15.104:0/125746865 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1940537897 0 0) 0x7fd6d402c820 con 0x7fd6d40327d0 2024-03-29T11:25:20.248 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.261+0000 7fd6d9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:20.248 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.261+0000 7fd6d9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:20.248 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.261+0000 7fd6d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:20.248 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.261+0000 7fd6d9ffb640 10 monclient(hunting): my global_id is 4455 2024-03-29T11:25:20.248 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.261+0000 7fd6d9ffb640 1 -- 172.21.15.104:0/125746865 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6d402a120 con 0x7fd6d40327d0 2024-03-29T11:25:20.249 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.262+0000 7fd6d9ffb640 1 -- 172.21.15.104:0/125746865 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2329521011 0 0) 0x7fd6c4003520 con 0x7fd6d4006410 2024-03-29T11:25:20.249 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.262+0000 7fd6d9ffb640 1 -- 172.21.15.104:0/125746865 >> v1:172.21.15.104:6790/0 conn(0x7fd6d402a640 legacy=0x7fd6d40320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:20.249 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.262+0000 7fd6d9ffb640 1 -- 172.21.15.104:0/125746865 >> v1:172.21.15.104:6789/0 conn(0x7fd6d40327d0 legacy=0x7fd6d40c8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:20.249 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.262+0000 7fd6d9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:25:20.249 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.262+0000 7fd6d9ffb640 1 monclient: found mon.noname-c 2024-03-29T11:25:20.249 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.262+0000 7fd6d9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:20.249 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.262+0000 7fd6d9ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.119:6789/0 2024-03-29T11:25:20.249 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.262+0000 7fd6d9ffb640 1 -- 172.21.15.104:0/125746865 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6d4033110 con 0x7fd6d4006410 2024-03-29T11:25:20.249 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.262+0000 7fd6d9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:25:20.249 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.262+0000 7fd6d9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:25:20.249 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.262+0000 7fd6d9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:20.249 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.262+0000 7fd6dbfff640 5 monclient: authenticate success, global_id 4426 2024-03-29T11:25:20.249 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.262+0000 7fd6dbfff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:20.249 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.262+0000 7fd6d9ffb640 1 -- 172.21.15.104:0/125746865 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fd6dc04f730 con 0x7fd6d4006410 2024-03-29T11:25:20.249 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.262+0000 7fd6d9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:20.250 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.262+0000 7fd6d9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:20.250 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.262+0000 7fd6d9ffb640 10 monclient: dump: 2024-03-29T11:25:20.250 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:25:20.250 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:25:20.250 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:20.250 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:20.250 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:20.250 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:25:20.250 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:25:20.250 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:25:20.250 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:25:20.250 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:25:20.250 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.262+0000 7fd6dbfff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:20.251 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.264+0000 7fd6d9ffb640 1 -- 172.21.15.104:0/125746865 <== mon.1 v1:172.21.15.119:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd6dc0008d0 con 0x7fd6d4006410 2024-03-29T11:25:20.251 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.264+0000 7fd6d9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:20.251 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.264+0000 7fd6d9ffb640 1 -- 172.21.15.104:0/125746865 <== mon.1 v1:172.21.15.119:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fd6dc0758b0 con 0x7fd6d4006410 2024-03-29T11:25:20.251 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.264+0000 7fd6d9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:20.251 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.264+0000 7fd6d9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:20.251 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.264+0000 7fd6d9ffb640 10 monclient: dump: 2024-03-29T11:25:20.251 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:25:20.251 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:25:20.251 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:20.251 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:20.252 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:20.252 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:25:20.252 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:25:20.252 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:25:20.252 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:25:20.252 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:25:20.252 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.265+0000 7fd6dbfff640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:20.252 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.265+0000 7fd6dbfff640 4 set_mon_vals no callback set 2024-03-29T11:25:20.252 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.265+0000 7fd6dbfff640 10 monclient: shutdown 2024-03-29T11:25:20.252 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.266+0000 7fd6dbfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:20.252 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.266+0000 7fd6dbfff640 1 -- 172.21.15.104:0/125746865 >> v1:172.21.15.119:6789/0 conn(0x7fd6d4006410 legacy=0x7fd6d40c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:20.252 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.266+0000 7fd6dbfff640 1 -- 172.21.15.104:0/125746865 shutdown_connections 2024-03-29T11:25:20.252 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.266+0000 7fd6dbfff640 1 -- 172.21.15.104:0/125746865 >> 172.21.15.104:0/125746865 conn(0x7fd6d40058f0 msgr2=0x7fd6d4005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:20.253 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.267+0000 7fd6dbfff640 1 -- 172.21.15.104:0/125746865 shutdown_connections 2024-03-29T11:25:20.257 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.268+0000 7fd6dbfff640 1 -- 172.21.15.104:0/125746865 wait complete. 2024-03-29T11:25:20.257 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.269+0000 7fd6dbfff640 10 monclient: build_initial_monmap 2024-03-29T11:25:20.257 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.269+0000 7fd6dbfff640 10 monclient: monmap: 2024-03-29T11:25:20.257 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:25:20.257 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:20.257 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:25:20.257 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:25:20.257 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:20.257 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:25:20.257 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:25:20.257 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:25:20.257 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:25:20.258 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:25:20.258 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.269+0000 7fd6dbfff640 1 librados: starting msgr at 2024-03-29T11:25:20.258 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.269+0000 7fd6dbfff640 1 librados: starting objecter 2024-03-29T11:25:20.258 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.269+0000 7fd6dbfff640 1 Processor -- start 2024-03-29T11:25:20.258 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.270+0000 7fd6dbfff640 1 -- start start 2024-03-29T11:25:20.258 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.270+0000 7fd6dbfff640 1 librados: setting wanted keys 2024-03-29T11:25:20.258 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.270+0000 7fd6dbfff640 1 librados: calling monclient init 2024-03-29T11:25:20.258 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.270+0000 7fd6dbfff640 10 monclient: init 2024-03-29T11:25:20.258 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.270+0000 7fd6dbfff640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:20.258 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.270+0000 7fd6dbfff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:25:20.258 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.270+0000 7fd6dbfff640 10 monclient(hunting): picked mon.noname-c con 0x7fd6d402a640 addr v1:172.21.15.104:6790/0 2024-03-29T11:25:20.258 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.270+0000 7fd6dbfff640 10 monclient(hunting): picked mon.noname-a con 0x7fd6d40327d0 addr v1:172.21.15.104:6789/0 2024-03-29T11:25:20.258 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.270+0000 7fd6dbfff640 10 monclient(hunting): picked mon.noname-b con 0x7fd6d40bae70 addr v1:172.21.15.119:6789/0 2024-03-29T11:25:20.258 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.270+0000 7fd6dbfff640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6d40d9a60 con 0x7fd6d40327d0 2024-03-29T11:25:20.258 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.270+0000 7fd6dbfff640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6d40da510 con 0x7fd6d40bae70 2024-03-29T11:25:20.261 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.270+0000 7fd6dbfff640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6d402cd70 con 0x7fd6d402a640 2024-03-29T11:25:20.262 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.270+0000 7fd6dbfff640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:20.262 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.270+0000 7fd6dbfff640 10 monclient(hunting): authenticate will time out at 1301.522805s 2024-03-29T11:25:20.267 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.275+0000 7fd6daffd640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7fd6d402a640 0x7fd6d40c42f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:45053/0 (socket says 172.21.15.104:45053) 2024-03-29T11:25:20.267 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.275+0000 7fd6daffd640 1 -- 172.21.15.104:0/1905434606 learned_addr learned my addr 172.21.15.104:0/1905434606 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:25:20.267 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.275+0000 7fd6c37fe640 10 client.?.objecter ms_handle_connect 0x7fd6d402a640 2024-03-29T11:25:20.267 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.275+0000 7fd6c37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:20.267 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.276+0000 7fd6c37fe640 1 -- 172.21.15.104:0/1905434606 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3977294301 0 0) 0x7fd6d402cd70 con 0x7fd6d402a640 2024-03-29T11:25:20.267 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.276+0000 7fd6c37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:25:20.267 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.276+0000 7fd6c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:20.268 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.276+0000 7fd6c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:20.268 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.276+0000 7fd6c37fe640 10 monclient(hunting): my global_id is 4457 2024-03-29T11:25:20.268 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.276+0000 7fd6c37fe640 1 -- 172.21.15.104:0/1905434606 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6c8004eb0 con 0x7fd6d402a640 2024-03-29T11:25:20.268 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.276+0000 7fd6c37fe640 10 client.?.objecter ms_handle_connect 0x7fd6d40327d0 2024-03-29T11:25:20.268 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.276+0000 7fd6c37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:20.268 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.276+0000 7fd6c37fe640 1 -- 172.21.15.104:0/1905434606 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2198638175 0 0) 0x7fd6d40d9a60 con 0x7fd6d40327d0 2024-03-29T11:25:20.268 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.276+0000 7fd6c37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:25:20.268 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.276+0000 7fd6c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:20.268 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.276+0000 7fd6c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:20.268 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.276+0000 7fd6c37fe640 10 monclient(hunting): my global_id is 4458 2024-03-29T11:25:20.268 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.276+0000 7fd6c37fe640 1 -- 172.21.15.104:0/1905434606 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6d402cd70 con 0x7fd6d40327d0 2024-03-29T11:25:20.268 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.277+0000 7fd6c37fe640 1 -- 172.21.15.104:0/1905434606 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2260448202 0 0) 0x7fd6d402cd70 con 0x7fd6d40327d0 2024-03-29T11:25:20.268 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.277+0000 7fd6c37fe640 1 -- 172.21.15.104:0/1905434606 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd6d40d9a60 con 0x7fd6d40327d0 2024-03-29T11:25:20.268 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.277+0000 7fd6c37fe640 1 -- 172.21.15.104:0/1905434606 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fd6d0001930 con 0x7fd6d40327d0 2024-03-29T11:25:20.269 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.277+0000 7fd6c37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:20.269 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.277+0000 7fd6c37fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:20.269 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.277+0000 7fd6c37fe640 10 monclient(hunting): dump: 2024-03-29T11:25:20.269 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:25:20.269 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:25:20.269 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:20.269 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:20.269 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:20.269 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:25:20.269 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:25:20.269 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:25:20.269 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:25:20.269 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:25:20.269 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.277+0000 7fd6c37fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:20.269 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.277+0000 7fd6c37fe640 1 -- 172.21.15.104:0/1905434606 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1405124771 0 0) 0x7fd6c8004eb0 con 0x7fd6d402a640 2024-03-29T11:25:20.269 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.277+0000 7fd6c37fe640 1 -- 172.21.15.104:0/1905434606 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd6d402cd70 con 0x7fd6d402a640 2024-03-29T11:25:20.270 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.277+0000 7fd6c37fe640 1 -- 172.21.15.104:0/1905434606 <== mon.0 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 360953597 0 0) 0x7fd6d40d9a60 con 0x7fd6d40327d0 2024-03-29T11:25:20.270 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.277+0000 7fd6c37fe640 1 -- 172.21.15.104:0/1905434606 >> v1:172.21.15.104:6790/0 conn(0x7fd6d402a640 legacy=0x7fd6d40c42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:20.270 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.277+0000 7fd6c37fe640 1 -- 172.21.15.104:0/1905434606 >> v1:172.21.15.119:6789/0 conn(0x7fd6d40bae70 legacy=0x7fd6d41636e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:25:20.270 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.278+0000 7fd6c37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:25:20.270 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.278+0000 7fd6c37fe640 1 monclient: found mon.a 2024-03-29T11:25:20.270 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.278+0000 7fd6c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:20.270 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.278+0000 7fd6c37fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:25:20.270 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.278+0000 7fd6c37fe640 1 -- 172.21.15.104:0/1905434606 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6d40ba950 con 0x7fd6d40327d0 2024-03-29T11:25:20.270 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.278+0000 7fd6c37fe640 10 monclient: _finish_auth 0 2024-03-29T11:25:20.270 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.278+0000 7fd6c37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:25:20.270 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.278+0000 7fd6c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:20.270 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.279+0000 7fd6dbfff640 5 monclient: authenticate success, global_id 4458 2024-03-29T11:25:20.270 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.279+0000 7fd6dbfff640 10 monclient: _renew_subs 2024-03-29T11:25:20.270 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.279+0000 7fd6dbfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:25:20.271 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.279+0000 7fd6dbfff640 1 -- 172.21.15.104:0/1905434606 --> v1:172.21.15.104:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6d40b7380 con 0x7fd6d40327d0 2024-03-29T11:25:20.271 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.279+0000 7fd6dbfff640 10 client.4458.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:20.271 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.279+0000 7fd6dbfff640 10 monclient: _renew_subs 2024-03-29T11:25:20.271 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.279+0000 7fd6dbfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:25:20.271 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.279+0000 7fd6dbfff640 1 -- 172.21.15.104:0/1905434606 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6d40ba3d0 con 0x7fd6d40327d0 2024-03-29T11:25:20.271 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.279+0000 7fd6dbfff640 1 librados: init done 2024-03-29T11:25:20.271 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.280+0000 7fd6c37fe640 1 -- 172.21.15.104:0/1905434606 <== mon.0 v1:172.21.15.104:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd6d0001a90 con 0x7fd6d40327d0 2024-03-29T11:25:20.271 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.280+0000 7fd6c37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:20.271 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.280+0000 7fd6c37fe640 1 -- 172.21.15.104:0/1905434606 <== mon.0 v1:172.21.15.104:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fd6d0003870 con 0x7fd6d40327d0 2024-03-29T11:25:20.271 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.280+0000 7fd6c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:20.271 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.280+0000 7fd6c37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:20.271 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.281+0000 7fd6d97fa640 4 set_mon_vals no callback set 2024-03-29T11:25:20.271 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.281+0000 7fd6c37fe640 10 monclient: dump: 2024-03-29T11:25:20.271 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:25:20.271 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:25:20.271 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:20.272 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:20.272 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:20.272 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:25:20.272 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:25:20.272 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:25:20.272 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:25:20.272 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:25:20.272 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.282+0000 7fd6dbfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:20.272 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.282+0000 7fd6dbfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:20.272 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.282+0000 7fd6dbfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:25:20.272 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.282+0000 7fd6dbfff640 1 -- 172.21.15.104:0/1905434606 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6bc005080 con 0x7fd6d40327d0 2024-03-29T11:25:20.273 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.285+0000 7fd6c37fe640 1 -- 172.21.15.104:0/1905434606 <== mon.0 v1:172.21.15.104:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7fd6d0019960 con 0x7fd6d40327d0 2024-03-29T11:25:20.273 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.286+0000 7fd6c37fe640 1 -- 172.21.15.104:0/1905434606 <== mon.0 v1:172.21.15.104:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7fd6d0083850 con 0x7fd6d40327d0 2024-03-29T11:25:20.273 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.286+0000 7fd6c37fe640 10 client.4458.objecter ms_dispatch 0x7fd6dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:20.274 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.286+0000 7fd6c37fe640 3 client.4458.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:20.274 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.286+0000 7fd6c37fe640 3 client.4458.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:20.274 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.286+0000 7fd6c37fe640 20 client.4458.objecter dump_active .. 0 homeless 2024-03-29T11:25:20.274 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.286+0000 7fd6daffd640 10 monclient: get_auth_request con 0x7fd6c806dcb0 auth_method 0 2024-03-29T11:25:20.274 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.287+0000 7fd6c37fe640 10 client.4458.objecter ms_handle_connect 0x7fd6c806dcb0 2024-03-29T11:25:20.277 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.292+0000 7fd6c37fe640 1 -- 172.21.15.104:0/1905434606 <== mon.0 v1:172.21.15.104:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd6d004f130 con 0x7fd6d40327d0 2024-03-29T11:25:20.277 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.292+0000 7fd6c37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:20.277 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.292+0000 7fd6c37fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:20.602 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.618+0000 7fd6dbfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:20.603 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.618+0000 7fd6dbfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:20.603 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.618+0000 7fd6dbfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:25:20.603 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.618+0000 7fd6dbfff640 1 -- 172.21.15.104:0/1905434606 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd6bc005330 con 0x7fd6d40327d0 2024-03-29T11:25:20.604 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.619+0000 7fd6c37fe640 1 -- 172.21.15.104:0/1905434606 <== mon.0 v1:172.21.15.104:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7fd6d0051060 con 0x7fd6d40327d0 2024-03-29T11:25:20.604 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.619+0000 7fd6c37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:20.604 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.619+0000 7fd6c37fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:20.607 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:25:20.607 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:20.609 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.625+0000 7fd6dbfff640 10 librados: watch_flush enter 2024-03-29T11:25:20.609 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.625+0000 7fd6dbfff640 10 librados: watch_flush exit 2024-03-29T11:25:20.610 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.625+0000 7fd6dbfff640 20 client.4458.objecter shutdown clearing up homeless session... 2024-03-29T11:25:20.610 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.625+0000 7fd6dbfff640 10 client.4458.objecter successfully canceled tick 2024-03-29T11:25:20.613 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.625+0000 7fd6dbfff640 1 -- 172.21.15.104:0/1905434606 >> v1:172.21.15.104:6816/3341126660 conn(0x7fd6c806dcb0 legacy=0x7fd6c8070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:20.613 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.625+0000 7fd6dbfff640 10 monclient: shutdown 2024-03-29T11:25:20.613 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.625+0000 7fd6dbfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:20.613 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.625+0000 7fd6dbfff640 1 -- 172.21.15.104:0/1905434606 >> v1:172.21.15.104:6789/0 conn(0x7fd6d40327d0 legacy=0x7fd6d40b6c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:20.613 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.626+0000 7fd6dbfff640 1 -- 172.21.15.104:0/1905434606 shutdown_connections 2024-03-29T11:25:20.613 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.626+0000 7fd6dbfff640 1 -- 172.21.15.104:0/1905434606 >> 172.21.15.104:0/1905434606 conn(0x7fd6d40058f0 msgr2=0x7fd6d40043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:20.617 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.632+0000 7fd6dbfff640 1 -- 172.21.15.104:0/1905434606 shutdown_connections 2024-03-29T11:25:20.617 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.632+0000 7fd6dbfff640 1 -- 172.21.15.104:0/1905434606 wait complete. 2024-03-29T11:25:20.617 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:20.632+0000 7fd6dbfff640 1 librados: shutdown 2024-03-29T11:25:20.634 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:27.442 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:27.594 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.608+0000 7f71e05c8640 10 monclient: get_monmap_and_config 2024-03-29T11:25:27.594 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.608+0000 7f71e05c8640 10 monclient: build_initial_monmap 2024-03-29T11:25:27.594 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.608+0000 7f71e05c8640 10 monclient: monmap: 2024-03-29T11:25:27.594 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:25:27.594 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:25:27.595 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:25:27.610088+0000 2024-03-29T11:25:27.595 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:25:27.610088+0000 2024-03-29T11:25:27.595 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:27.595 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:25:27.595 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:25:27.595 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:25:27.595 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:25:27.595 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:25:27.595 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.609+0000 7f71e05c8640 1 Processor -- start 2024-03-29T11:25:27.595 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.609+0000 7f71e05c8640 1 -- start start 2024-03-29T11:25:27.595 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.609+0000 7f71e05c8640 10 monclient: init 2024-03-29T11:25:27.597 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.610+0000 7f71e05c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:27.597 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.610+0000 7f71e05c8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:25:27.597 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.610+0000 7f71e05c8640 10 monclient(hunting): picked mon.noname-a con 0x7f71d80a96c0 addr v1:172.21.15.104:6789/0 2024-03-29T11:25:27.597 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.610+0000 7f71e05c8640 10 monclient(hunting): picked mon.noname-b con 0x7f71d80a9aa0 addr v1:172.21.15.104:6790/0 2024-03-29T11:25:27.597 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.610+0000 7f71e05c8640 10 monclient(hunting): picked mon.noname-c con 0x7f71d80ad730 addr v1:172.21.15.119:6789/0 2024-03-29T11:25:27.597 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.610+0000 7f71e05c8640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71d813fee0 con 0x7f71d80a96c0 2024-03-29T11:25:27.597 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.610+0000 7f71e05c8640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71d80aab80 con 0x7f71d80ad730 2024-03-29T11:25:27.597 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.610+0000 7f71e05c8640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71d8145e80 con 0x7f71d80a9aa0 2024-03-29T11:25:27.597 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.610+0000 7f71e05c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:27.597 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.610+0000 7f71e05c8640 10 monclient(hunting): authenticate will time out at 1308.862387s 2024-03-29T11:25:27.599 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.614+0000 7f71deb65640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f71d80ad730 0x7f71d814dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:59209/0 (socket says 172.21.15.104:59209) 2024-03-29T11:25:27.599 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.614+0000 7f71deb65640 1 -- 172.21.15.104:0/1285856567 learned_addr learned my addr 172.21.15.104:0/1285856567 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:25:27.602 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.615+0000 7f71dd362640 1 -- 172.21.15.104:0/1285856567 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3956142132 0 0) 0x7f71d80aab80 con 0x7f71d80ad730 2024-03-29T11:25:27.603 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.615+0000 7f71dd362640 10 monclient(hunting): _negotiate 2024-03-29T11:25:27.603 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.615+0000 7f71dd362640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:27.603 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.615+0000 7f71dd362640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:27.603 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.615+0000 7f71dd362640 10 monclient(hunting): my global_id is 4429 2024-03-29T11:25:27.603 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.615+0000 7f71dd362640 1 -- 172.21.15.104:0/1285856567 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71c4003520 con 0x7f71d80ad730 2024-03-29T11:25:27.603 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.615+0000 7f71dd362640 1 -- 172.21.15.104:0/1285856567 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1245695383 0 0) 0x7f71d8145e80 con 0x7f71d80a9aa0 2024-03-29T11:25:27.603 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.615+0000 7f71dd362640 10 monclient(hunting): _negotiate 2024-03-29T11:25:27.603 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.615+0000 7f71dd362640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:27.603 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.615+0000 7f71dd362640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:27.603 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.615+0000 7f71dd362640 10 monclient(hunting): my global_id is 4460 2024-03-29T11:25:27.603 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.615+0000 7f71dd362640 1 -- 172.21.15.104:0/1285856567 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71d80aab80 con 0x7f71d80a9aa0 2024-03-29T11:25:27.603 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.616+0000 7f71dd362640 1 -- 172.21.15.104:0/1285856567 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 102332153 0 0) 0x7f71d80aab80 con 0x7f71d80a9aa0 2024-03-29T11:25:27.603 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.616+0000 7f71dd362640 1 -- 172.21.15.104:0/1285856567 >> v1:172.21.15.119:6789/0 conn(0x7f71d80ad730 legacy=0x7f71d814dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:27.604 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.616+0000 7f71dd362640 1 -- 172.21.15.104:0/1285856567 >> v1:172.21.15.104:6789/0 conn(0x7f71d80a96c0 legacy=0x7f71d813f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:27.604 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.616+0000 7f71dd362640 10 monclient: _finish_hunting 0 2024-03-29T11:25:27.604 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.616+0000 7f71dd362640 1 monclient: found mon.noname-b 2024-03-29T11:25:27.604 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.616+0000 7f71dd362640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:27.604 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.616+0000 7f71dd362640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.104:6790/0 2024-03-29T11:25:27.604 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.616+0000 7f71dd362640 1 -- 172.21.15.104:0/1285856567 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71d80b4440 con 0x7f71d80a9aa0 2024-03-29T11:25:27.604 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.616+0000 7f71dd362640 10 monclient: _finish_auth 0 2024-03-29T11:25:27.604 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.616+0000 7f71dd362640 10 monclient: _check_auth_tickets 2024-03-29T11:25:27.604 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.616+0000 7f71dd362640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:27.604 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.616+0000 7f71dd362640 1 -- 172.21.15.104:0/1285856567 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f71d4001670 con 0x7f71d80a9aa0 2024-03-29T11:25:27.604 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.616+0000 7f71dd362640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:27.604 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.616+0000 7f71dd362640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:27.604 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.616+0000 7f71dd362640 10 monclient: dump: 2024-03-29T11:25:27.604 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:25:27.605 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:25:27.605 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:27.605 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:27.605 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:27.605 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:25:27.605 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:25:27.605 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:25:27.605 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:25:27.605 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:25:27.605 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.616+0000 7f71e05c8640 5 monclient: authenticate success, global_id 4460 2024-03-29T11:25:27.605 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.616+0000 7f71e05c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:27.605 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.616+0000 7f71dd362640 1 -- 172.21.15.104:0/1285856567 <== mon.2 v1:172.21.15.104:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f71d40017d0 con 0x7f71d80a9aa0 2024-03-29T11:25:27.605 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.616+0000 7f71dd362640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:27.606 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.616+0000 7f71dd362640 1 -- 172.21.15.104:0/1285856567 <== mon.2 v1:172.21.15.104:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f71d4001be0 con 0x7f71d80a9aa0 2024-03-29T11:25:27.606 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.616+0000 7f71dd362640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:27.606 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.616+0000 7f71dd362640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:27.606 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.616+0000 7f71dd362640 10 monclient: dump: 2024-03-29T11:25:27.606 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:25:27.606 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:25:27.606 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:27.606 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:27.606 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:27.606 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:25:27.606 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:25:27.606 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:25:27.606 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:25:27.607 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:25:27.607 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.616+0000 7f71e05c8640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:27.607 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.616+0000 7f71e05c8640 4 set_mon_vals no callback set 2024-03-29T11:25:27.607 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.617+0000 7f71e05c8640 10 monclient: shutdown 2024-03-29T11:25:27.607 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.617+0000 7f71e05c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:27.607 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.617+0000 7f71e05c8640 1 -- 172.21.15.104:0/1285856567 >> v1:172.21.15.104:6790/0 conn(0x7f71d80a9aa0 legacy=0x7f71d8145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:27.607 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.618+0000 7f71e05c8640 1 -- 172.21.15.104:0/1285856567 shutdown_connections 2024-03-29T11:25:27.607 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.618+0000 7f71e05c8640 1 -- 172.21.15.104:0/1285856567 >> 172.21.15.104:0/1285856567 conn(0x7f71d80b35e0 msgr2=0x7f71d80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:27.608 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.621+0000 7f71e05c8640 1 -- 172.21.15.104:0/1285856567 shutdown_connections 2024-03-29T11:25:27.608 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.621+0000 7f71e05c8640 1 -- 172.21.15.104:0/1285856567 wait complete. 2024-03-29T11:25:27.608 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.622+0000 7f71e05c8640 10 monclient: build_initial_monmap 2024-03-29T11:25:27.608 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.622+0000 7f71e05c8640 10 monclient: monmap: 2024-03-29T11:25:27.608 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:25:27.608 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:27.608 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:25:27.608 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:25:27.609 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:27.609 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:25:27.609 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:25:27.609 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:25:27.609 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:25:27.609 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:25:27.609 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.622+0000 7f71e05c8640 1 librados: starting msgr at 2024-03-29T11:25:27.609 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.622+0000 7f71e05c8640 1 librados: starting objecter 2024-03-29T11:25:27.609 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.622+0000 7f71e05c8640 1 Processor -- start 2024-03-29T11:25:27.609 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.622+0000 7f71e05c8640 1 -- start start 2024-03-29T11:25:27.609 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.622+0000 7f71e05c8640 1 librados: setting wanted keys 2024-03-29T11:25:27.609 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.622+0000 7f71e05c8640 1 librados: calling monclient init 2024-03-29T11:25:27.609 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.622+0000 7f71e05c8640 10 monclient: init 2024-03-29T11:25:27.609 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.623+0000 7f71e05c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:27.609 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.623+0000 7f71e05c8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:25:27.610 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.624+0000 7f71e05c8640 10 monclient(hunting): picked mon.noname-b con 0x7f71d80ad730 addr v1:172.21.15.119:6789/0 2024-03-29T11:25:27.610 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.624+0000 7f71e05c8640 10 monclient(hunting): picked mon.noname-a con 0x7f71d806e7a0 addr v1:172.21.15.104:6789/0 2024-03-29T11:25:27.610 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.624+0000 7f71e05c8640 10 monclient(hunting): picked mon.noname-c con 0x7f71d81cb280 addr v1:172.21.15.104:6790/0 2024-03-29T11:25:27.610 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.624+0000 7f71e05c8640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71d81cb8b0 con 0x7f71d806e7a0 2024-03-29T11:25:27.610 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.624+0000 7f71e05c8640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71d806fa00 con 0x7f71d80ad730 2024-03-29T11:25:27.610 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.624+0000 7f71e05c8640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71d81ce900 con 0x7f71d81cb280 2024-03-29T11:25:27.610 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.624+0000 7f71e05c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:27.610 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.624+0000 7f71e05c8640 10 monclient(hunting): authenticate will time out at 1308.876391s 2024-03-29T11:25:27.613 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.628+0000 7f71de364640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f71d80ad730 0x7f71d81cbd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:59253/0 (socket says 172.21.15.104:59253) 2024-03-29T11:25:27.613 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.628+0000 7f71de364640 1 -- 172.21.15.104:0/152476491 learned_addr learned my addr 172.21.15.104:0/152476491 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:25:27.615 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.630+0000 7f71ceffd640 10 client.?.objecter ms_handle_connect 0x7f71d80ad730 2024-03-29T11:25:27.615 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.630+0000 7f71ceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:27.615 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.630+0000 7f71ceffd640 1 -- 172.21.15.104:0/152476491 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 369927073 0 0) 0x7f71d806fa00 con 0x7f71d80ad730 2024-03-29T11:25:27.615 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.630+0000 7f71ceffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:27.615 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.630+0000 7f71ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:27.616 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.630+0000 7f71ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:27.616 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.630+0000 7f71ceffd640 10 monclient(hunting): my global_id is 4432 2024-03-29T11:25:27.616 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.630+0000 7f71ceffd640 1 -- 172.21.15.104:0/152476491 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71bc004eb0 con 0x7f71d80ad730 2024-03-29T11:25:27.619 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.633+0000 7f71ceffd640 1 -- 172.21.15.104:0/152476491 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 783409681 0 0) 0x7f71bc004eb0 con 0x7f71d80ad730 2024-03-29T11:25:27.619 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.633+0000 7f71ceffd640 1 -- 172.21.15.104:0/152476491 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71d806fa00 con 0x7f71d80ad730 2024-03-29T11:25:27.619 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.634+0000 7f71ceffd640 10 client.?.objecter ms_handle_connect 0x7f71d806e7a0 2024-03-29T11:25:27.619 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.634+0000 7f71ceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:27.620 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.634+0000 7f71ceffd640 1 -- 172.21.15.104:0/152476491 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f71d0001d20 con 0x7f71d80ad730 2024-03-29T11:25:27.620 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.634+0000 7f71ceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:27.620 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.634+0000 7f71ceffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:27.620 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.634+0000 7f71ceffd640 10 monclient(hunting): dump: 2024-03-29T11:25:27.620 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:25:27.620 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:25:27.620 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:27.620 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:27.620 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:27.620 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:25:27.620 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:25:27.620 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:25:27.620 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:25:27.620 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:25:27.620 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.634+0000 7f71ceffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:27.625 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.637+0000 7f71ceffd640 10 client.?.objecter ms_handle_connect 0x7f71d81cb280 2024-03-29T11:25:27.625 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.637+0000 7f71ceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:27.625 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.637+0000 7f71ceffd640 1 -- 172.21.15.104:0/152476491 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1984263976 0 0) 0x7f71d81ce900 con 0x7f71d81cb280 2024-03-29T11:25:27.625 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.638+0000 7f71ceffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:27.625 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.638+0000 7f71ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:27.625 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.638+0000 7f71ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:27.625 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.638+0000 7f71ceffd640 10 monclient(hunting): my global_id is 4463 2024-03-29T11:25:27.625 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.638+0000 7f71ceffd640 1 -- 172.21.15.104:0/152476491 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71bc004eb0 con 0x7f71d81cb280 2024-03-29T11:25:27.625 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.638+0000 7f71ceffd640 1 -- 172.21.15.104:0/152476491 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4019615748 0 0) 0x7f71d806fa00 con 0x7f71d80ad730 2024-03-29T11:25:27.625 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.638+0000 7f71ceffd640 1 -- 172.21.15.104:0/152476491 >> v1:172.21.15.104:6790/0 conn(0x7f71d81cb280 legacy=0x7f71d81e1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:27.625 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.638+0000 7f71ceffd640 1 -- 172.21.15.104:0/152476491 >> v1:172.21.15.104:6789/0 conn(0x7f71d806e7a0 legacy=0x7f71d81cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:27.625 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.638+0000 7f71ceffd640 10 monclient: _finish_hunting 0 2024-03-29T11:25:27.626 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.638+0000 7f71ceffd640 1 monclient: found mon.b 2024-03-29T11:25:27.626 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.638+0000 7f71ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:27.626 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.638+0000 7f71ceffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:25:27.626 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.638+0000 7f71ceffd640 1 -- 172.21.15.104:0/152476491 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71d81cea90 con 0x7f71d80ad730 2024-03-29T11:25:27.626 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.638+0000 7f71ceffd640 10 monclient: _finish_auth 0 2024-03-29T11:25:27.626 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.638+0000 7f71ceffd640 10 monclient: _check_auth_tickets 2024-03-29T11:25:27.626 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.638+0000 7f71ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:27.626 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.638+0000 7f71e05c8640 5 monclient: authenticate success, global_id 4432 2024-03-29T11:25:27.626 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.638+0000 7f71e05c8640 10 monclient: _renew_subs 2024-03-29T11:25:27.626 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.638+0000 7f71e05c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:25:27.626 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.638+0000 7f71e05c8640 1 -- 172.21.15.104:0/152476491 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71d81cece0 con 0x7f71d80ad730 2024-03-29T11:25:27.626 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.639+0000 7f71e05c8640 10 client.4432.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:27.626 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.639+0000 7f71e05c8640 10 monclient: _renew_subs 2024-03-29T11:25:27.626 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.639+0000 7f71e05c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:25:27.626 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.639+0000 7f71e05c8640 1 -- 172.21.15.104:0/152476491 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f71d81e35a0 con 0x7f71d80ad730 2024-03-29T11:25:27.627 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.639+0000 7f71e05c8640 1 librados: init done 2024-03-29T11:25:27.627 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.640+0000 7f71ceffd640 1 -- 172.21.15.104:0/152476491 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f71d00020e0 con 0x7f71d80ad730 2024-03-29T11:25:27.627 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.640+0000 7f71ceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:27.627 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.640+0000 7f71dcb61640 4 set_mon_vals no callback set 2024-03-29T11:25:27.627 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.641+0000 7f71ceffd640 1 -- 172.21.15.104:0/152476491 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f71d0003c40 con 0x7f71d80ad730 2024-03-29T11:25:27.628 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.641+0000 7f71ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:27.629 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.643+0000 7f71ceffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:27.629 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.643+0000 7f71ceffd640 10 monclient: dump: 2024-03-29T11:25:27.629 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:25:27.629 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:25:27.629 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:27.629 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:27.629 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:27.629 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:25:27.629 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:25:27.629 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:25:27.629 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:25:27.629 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:25:27.630 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.643+0000 7f71ceffd640 1 -- 172.21.15.104:0/152476491 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f71d0003e60 con 0x7f71d80ad730 2024-03-29T11:25:27.630 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.643+0000 7f71ceffd640 1 -- 172.21.15.104:0/152476491 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f71d0084e00 con 0x7f71d80ad730 2024-03-29T11:25:27.630 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.643+0000 7f71ceffd640 10 client.4432.objecter ms_dispatch 0x7f71d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:27.630 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.643+0000 7f71ceffd640 3 client.4432.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:27.630 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.643+0000 7f71ceffd640 3 client.4432.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:27.630 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.643+0000 7f71ceffd640 20 client.4432.objecter dump_active .. 0 homeless 2024-03-29T11:25:27.630 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.644+0000 7f71ddb63640 10 monclient: get_auth_request con 0x7f71bc06dbf0 auth_method 0 2024-03-29T11:25:27.631 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.644+0000 7f71e05c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:27.631 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.644+0000 7f71e05c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:27.631 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.644+0000 7f71e05c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:25:27.631 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.644+0000 7f71e05c8640 1 -- 172.21.15.104:0/152476491 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71d8075f10 con 0x7f71d80ad730 2024-03-29T11:25:27.631 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.644+0000 7f71ceffd640 10 client.4432.objecter ms_handle_connect 0x7f71bc06dbf0 2024-03-29T11:25:27.638 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.653+0000 7f71ceffd640 1 -- 172.21.15.104:0/152476491 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f71d00506e0 con 0x7f71d80ad730 2024-03-29T11:25:27.638 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.653+0000 7f71ceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:27.638 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.653+0000 7f71ceffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:27.961 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.976+0000 7f71e05c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:27.961 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.976+0000 7f71e05c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:27.961 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.976+0000 7f71e05c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:25:27.961 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.976+0000 7f71e05c8640 1 -- 172.21.15.104:0/152476491 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f71d80b27b0 con 0x7f71d80ad730 2024-03-29T11:25:27.962 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.977+0000 7f71ceffd640 1 -- 172.21.15.104:0/152476491 <== mon.1 v1:172.21.15.119:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f71d0052610 con 0x7f71d80ad730 2024-03-29T11:25:27.962 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.977+0000 7f71ceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:27.962 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.977+0000 7f71ceffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:27.962 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:25:27.962 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:27.965 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.980+0000 7f71e05c8640 10 librados: watch_flush enter 2024-03-29T11:25:27.966 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.980+0000 7f71e05c8640 10 librados: watch_flush exit 2024-03-29T11:25:27.966 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.980+0000 7f71e05c8640 20 client.4432.objecter shutdown clearing up homeless session... 2024-03-29T11:25:27.966 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.980+0000 7f71e05c8640 10 client.4432.objecter successfully canceled tick 2024-03-29T11:25:27.966 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.981+0000 7f71e05c8640 1 -- 172.21.15.104:0/152476491 >> v1:172.21.15.104:6816/3341126660 conn(0x7f71bc06dbf0 legacy=0x7f71bc0700b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:27.966 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.981+0000 7f71e05c8640 10 monclient: shutdown 2024-03-29T11:25:27.966 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.981+0000 7f71e05c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:27.966 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.981+0000 7f71e05c8640 1 -- 172.21.15.104:0/152476491 >> v1:172.21.15.119:6789/0 conn(0x7f71d80ad730 legacy=0x7f71d81cbd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:27.967 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.982+0000 7f71e05c8640 1 -- 172.21.15.104:0/152476491 shutdown_connections 2024-03-29T11:25:27.967 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.982+0000 7f71e05c8640 1 -- 172.21.15.104:0/152476491 >> 172.21.15.104:0/152476491 conn(0x7f71d80b35e0 msgr2=0x7f71d80b0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:27.968 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.983+0000 7f71e05c8640 1 -- 172.21.15.104:0/152476491 shutdown_connections 2024-03-29T11:25:27.968 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.983+0000 7f71e05c8640 1 -- 172.21.15.104:0/152476491 wait complete. 2024-03-29T11:25:27.968 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:27.983+0000 7f71e05c8640 1 librados: shutdown 2024-03-29T11:25:27.984 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:34.792 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:34.938 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.953+0000 7f2e9e0d9640 10 monclient: get_monmap_and_config 2024-03-29T11:25:34.939 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.953+0000 7f2e9e0d9640 10 monclient: build_initial_monmap 2024-03-29T11:25:34.939 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.953+0000 7f2e9e0d9640 10 monclient: monmap: 2024-03-29T11:25:34.939 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:25:34.939 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:25:34.939 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:25:34.954403+0000 2024-03-29T11:25:34.939 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:25:34.954403+0000 2024-03-29T11:25:34.939 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:34.939 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:25:34.939 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:25:34.939 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:25:34.939 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:25:34.939 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:25:34.939 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.954+0000 7f2e9e0d9640 1 Processor -- start 2024-03-29T11:25:34.940 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.954+0000 7f2e9e0d9640 1 -- start start 2024-03-29T11:25:34.940 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.954+0000 7f2e9e0d9640 10 monclient: init 2024-03-29T11:25:34.940 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.955+0000 7f2e9e0d9640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:34.940 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.955+0000 7f2e9e0d9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:25:34.940 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.955+0000 7f2e9e0d9640 10 monclient(hunting): picked mon.noname-c con 0x7f2e980a9ac0 addr v1:172.21.15.119:6789/0 2024-03-29T11:25:34.940 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.956+0000 7f2e9e0d9640 10 monclient(hunting): picked mon.noname-b con 0x7f2e9813ee90 addr v1:172.21.15.104:6790/0 2024-03-29T11:25:34.940 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.956+0000 7f2e9e0d9640 10 monclient(hunting): picked mon.noname-a con 0x7f2e980af900 addr v1:172.21.15.104:6789/0 2024-03-29T11:25:34.941 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.956+0000 7f2e9e0d9640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e980afd10 con 0x7f2e980af900 2024-03-29T11:25:34.941 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.956+0000 7f2e9e0d9640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e980af6f0 con 0x7f2e980a9ac0 2024-03-29T11:25:34.941 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.956+0000 7f2e9e0d9640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e980aac80 con 0x7f2e9813ee90 2024-03-29T11:25:34.941 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.956+0000 7f2e9e0d9640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:34.941 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.956+0000 7f2e9e0d9640 10 monclient(hunting): authenticate will time out at 1316.207085s 2024-03-29T11:25:34.944 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.959+0000 7f2e977fe640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f2e980a9ac0 0x7f2e980adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:34537/0 (socket says 172.21.15.104:34537) 2024-03-29T11:25:34.944 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.959+0000 7f2e977fe640 1 -- 172.21.15.104:0/3523132240 learned_addr learned my addr 172.21.15.104:0/3523132240 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:25:34.945 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.960+0000 7f2e967fc640 1 -- 172.21.15.104:0/3523132240 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3663951008 0 0) 0x7f2e980af6f0 con 0x7f2e980a9ac0 2024-03-29T11:25:34.945 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.960+0000 7f2e967fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:34.945 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.960+0000 7f2e967fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:34.945 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.960+0000 7f2e967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:34.945 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.960+0000 7f2e967fc640 10 monclient(hunting): my global_id is 4435 2024-03-29T11:25:34.945 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.960+0000 7f2e967fc640 1 -- 172.21.15.104:0/3523132240 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e80003520 con 0x7f2e980a9ac0 2024-03-29T11:25:34.945 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.961+0000 7f2e967fc640 1 -- 172.21.15.104:0/3523132240 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1436716599 0 0) 0x7f2e80003520 con 0x7f2e980a9ac0 2024-03-29T11:25:34.945 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.961+0000 7f2e967fc640 1 -- 172.21.15.104:0/3523132240 >> v1:172.21.15.104:6790/0 conn(0x7f2e9813ee90 legacy=0x7f2e9813b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:34.948 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.963+0000 7f2e967fc640 1 -- 172.21.15.104:0/3523132240 >> v1:172.21.15.104:6789/0 conn(0x7f2e980af900 legacy=0x7f2e98149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:34.948 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.964+0000 7f2e967fc640 10 monclient: _finish_hunting 0 2024-03-29T11:25:34.948 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.964+0000 7f2e967fc640 1 monclient: found mon.noname-c 2024-03-29T11:25:34.948 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.964+0000 7f2e967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:34.948 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.964+0000 7f2e967fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.119:6789/0 2024-03-29T11:25:34.948 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.964+0000 7f2e967fc640 1 -- 172.21.15.104:0/3523132240 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e980aaf60 con 0x7f2e980a9ac0 2024-03-29T11:25:34.949 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.964+0000 7f2e967fc640 10 monclient: _finish_auth 0 2024-03-29T11:25:34.949 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.964+0000 7f2e967fc640 10 monclient: _check_auth_tickets 2024-03-29T11:25:34.949 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.964+0000 7f2e967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:34.949 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.964+0000 7f2e967fc640 1 -- 172.21.15.104:0/3523132240 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f2e88001690 con 0x7f2e980a9ac0 2024-03-29T11:25:34.949 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.964+0000 7f2e967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:34.949 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.964+0000 7f2e967fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:34.949 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.964+0000 7f2e967fc640 10 monclient: dump: 2024-03-29T11:25:34.949 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:25:34.949 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:25:34.949 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:34.949 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:34.949 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:34.949 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:25:34.949 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:25:34.949 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:25:34.950 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:25:34.950 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:25:34.951 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.964+0000 7f2e9e0d9640 5 monclient: authenticate success, global_id 4435 2024-03-29T11:25:34.951 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.964+0000 7f2e9e0d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:34.951 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.965+0000 7f2e967fc640 1 -- 172.21.15.104:0/3523132240 <== mon.1 v1:172.21.15.119:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e88001860 con 0x7f2e980a9ac0 2024-03-29T11:25:34.951 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.965+0000 7f2e967fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:34.951 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.965+0000 7f2e967fc640 1 -- 172.21.15.104:0/3523132240 <== mon.1 v1:172.21.15.119:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f2e88001c70 con 0x7f2e980a9ac0 2024-03-29T11:25:34.951 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.965+0000 7f2e967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:34.951 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.965+0000 7f2e967fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:34.951 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.965+0000 7f2e967fc640 10 monclient: dump: 2024-03-29T11:25:34.951 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:25:34.952 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:25:34.952 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:34.952 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:34.952 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:34.952 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:25:34.952 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:25:34.952 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:25:34.952 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:25:34.952 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:25:34.952 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.965+0000 7f2e9e0d9640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:34.952 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.965+0000 7f2e9e0d9640 4 set_mon_vals no callback set 2024-03-29T11:25:34.953 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.968+0000 7f2e9e0d9640 10 monclient: shutdown 2024-03-29T11:25:34.953 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.968+0000 7f2e9e0d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:34.953 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.968+0000 7f2e9e0d9640 1 -- 172.21.15.104:0/3523132240 >> v1:172.21.15.119:6789/0 conn(0x7f2e980a9ac0 legacy=0x7f2e980adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:34.957 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.973+0000 7f2e9e0d9640 1 -- 172.21.15.104:0/3523132240 shutdown_connections 2024-03-29T11:25:34.957 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.973+0000 7f2e9e0d9640 1 -- 172.21.15.104:0/3523132240 >> 172.21.15.104:0/3523132240 conn(0x7f2e980b35e0 msgr2=0x7f2e980b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:34.957 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.973+0000 7f2e9e0d9640 1 -- 172.21.15.104:0/3523132240 shutdown_connections 2024-03-29T11:25:34.957 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.973+0000 7f2e9e0d9640 1 -- 172.21.15.104:0/3523132240 wait complete. 2024-03-29T11:25:34.959 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.974+0000 7f2e9e0d9640 10 monclient: build_initial_monmap 2024-03-29T11:25:34.959 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.974+0000 7f2e9e0d9640 10 monclient: monmap: 2024-03-29T11:25:34.959 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:25:34.959 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:34.959 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:25:34.959 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:25:34.959 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:34.959 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:25:34.959 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:25:34.959 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:25:34.959 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:25:34.959 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:25:34.960 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.974+0000 7f2e9e0d9640 1 librados: starting msgr at 2024-03-29T11:25:34.960 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.974+0000 7f2e9e0d9640 1 librados: starting objecter 2024-03-29T11:25:34.960 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.974+0000 7f2e9e0d9640 1 Processor -- start 2024-03-29T11:25:34.964 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.975+0000 7f2e9e0d9640 1 -- start start 2024-03-29T11:25:34.964 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.975+0000 7f2e9e0d9640 1 librados: setting wanted keys 2024-03-29T11:25:34.964 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.975+0000 7f2e9e0d9640 1 librados: calling monclient init 2024-03-29T11:25:34.964 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.975+0000 7f2e9e0d9640 10 monclient: init 2024-03-29T11:25:34.964 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.975+0000 7f2e9e0d9640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:34.964 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.975+0000 7f2e9e0d9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:25:34.964 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.975+0000 7f2e9e0d9640 10 monclient(hunting): picked mon.noname-c con 0x7f2e980a9ac0 addr v1:172.21.15.104:6790/0 2024-03-29T11:25:34.964 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.975+0000 7f2e9e0d9640 10 monclient(hunting): picked mon.noname-a con 0x7f2e980af900 addr v1:172.21.15.104:6789/0 2024-03-29T11:25:34.964 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.975+0000 7f2e9e0d9640 10 monclient(hunting): picked mon.noname-b con 0x7f2e9813ee90 addr v1:172.21.15.119:6789/0 2024-03-29T11:25:34.964 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.975+0000 7f2e9e0d9640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e98141100 con 0x7f2e980af900 2024-03-29T11:25:34.965 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.975+0000 7f2e9e0d9640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e9813fc70 con 0x7f2e9813ee90 2024-03-29T11:25:34.965 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.975+0000 7f2e9e0d9640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e9813dfd0 con 0x7f2e980a9ac0 2024-03-29T11:25:34.965 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.975+0000 7f2e9e0d9640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:34.965 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.975+0000 7f2e9e0d9640 10 monclient(hunting): authenticate will time out at 1316.226980s 2024-03-29T11:25:34.965 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.978+0000 7f2e977fe640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f2e980a9ac0 0x7f2e9812ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:38243/0 (socket says 172.21.15.104:38243) 2024-03-29T11:25:34.965 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.979+0000 7f2e977fe640 1 -- 172.21.15.104:0/2735132388 learned_addr learned my addr 172.21.15.104:0/2735132388 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:25:34.966 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.982+0000 7f2e7ffff640 10 client.?.objecter ms_handle_connect 0x7f2e980a9ac0 2024-03-29T11:25:34.966 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.982+0000 7f2e7ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:34.966 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.982+0000 7f2e7ffff640 10 client.?.objecter ms_handle_connect 0x7f2e9813ee90 2024-03-29T11:25:34.967 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.982+0000 7f2e7ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:34.967 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.982+0000 7f2e7ffff640 1 -- 172.21.15.104:0/2735132388 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1389652080 0 0) 0x7f2e9813fc70 con 0x7f2e9813ee90 2024-03-29T11:25:34.967 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.982+0000 7f2e7ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:34.969 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.982+0000 7f2e7ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:34.969 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.982+0000 7f2e7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:34.969 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.982+0000 7f2e7ffff640 10 monclient(hunting): my global_id is 4438 2024-03-29T11:25:34.969 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.982+0000 7f2e7ffff640 1 -- 172.21.15.104:0/2735132388 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e84004eb0 con 0x7f2e9813ee90 2024-03-29T11:25:34.969 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.982+0000 7f2e7ffff640 1 -- 172.21.15.104:0/2735132388 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2110844370 0 0) 0x7f2e9813dfd0 con 0x7f2e980a9ac0 2024-03-29T11:25:34.969 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.982+0000 7f2e7ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:34.969 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.982+0000 7f2e7ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:34.969 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.982+0000 7f2e7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:34.969 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.982+0000 7f2e7ffff640 10 monclient(hunting): my global_id is 4466 2024-03-29T11:25:34.969 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.982+0000 7f2e7ffff640 1 -- 172.21.15.104:0/2735132388 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e9813fc70 con 0x7f2e980a9ac0 2024-03-29T11:25:34.969 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.983+0000 7f2e7ffff640 1 -- 172.21.15.104:0/2735132388 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2195732051 0 0) 0x7f2e9813fc70 con 0x7f2e980a9ac0 2024-03-29T11:25:34.970 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.983+0000 7f2e7ffff640 1 -- 172.21.15.104:0/2735132388 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e9813dfd0 con 0x7f2e980a9ac0 2024-03-29T11:25:34.970 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.983+0000 7f2e7ffff640 1 -- 172.21.15.104:0/2735132388 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f2e88001b70 con 0x7f2e980a9ac0 2024-03-29T11:25:34.970 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.983+0000 7f2e7ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:34.970 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.983+0000 7f2e7ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:34.970 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.983+0000 7f2e7ffff640 10 monclient(hunting): dump: 2024-03-29T11:25:34.970 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:25:34.970 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:25:34.970 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:34.970 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:34.970 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:34.970 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:25:34.970 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:25:34.970 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:25:34.971 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:25:34.971 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:25:34.971 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.983+0000 7f2e7ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:34.971 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.983+0000 7f2e7ffff640 10 client.?.objecter ms_handle_connect 0x7f2e980af900 2024-03-29T11:25:34.971 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.983+0000 7f2e7ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:34.973 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.985+0000 7f2e7ffff640 1 -- 172.21.15.104:0/2735132388 <== mon.2 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4040674043 0 0) 0x7f2e9813dfd0 con 0x7f2e980a9ac0 2024-03-29T11:25:34.973 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.985+0000 7f2e7ffff640 1 -- 172.21.15.104:0/2735132388 >> v1:172.21.15.119:6789/0 conn(0x7f2e9813ee90 legacy=0x7f2e981dbd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:34.973 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.985+0000 7f2e7ffff640 1 -- 172.21.15.104:0/2735132388 >> v1:172.21.15.104:6789/0 conn(0x7f2e980af900 legacy=0x7f2e981db620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:34.973 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.985+0000 7f2e7ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:25:34.973 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.985+0000 7f2e7ffff640 1 monclient: found mon.c 2024-03-29T11:25:34.973 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.985+0000 7f2e7ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:34.973 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.985+0000 7f2e7ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:25:34.973 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.985+0000 7f2e7ffff640 1 -- 172.21.15.104:0/2735132388 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e9813e160 con 0x7f2e980a9ac0 2024-03-29T11:25:34.973 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.985+0000 7f2e7ffff640 10 monclient: _finish_auth 0 2024-03-29T11:25:34.974 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.985+0000 7f2e7ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:25:34.974 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.985+0000 7f2e7ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:34.974 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.986+0000 7f2e7ffff640 1 -- 172.21.15.104:0/2735132388 <== mon.2 v1:172.21.15.104:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e88004bc0 con 0x7f2e980a9ac0 2024-03-29T11:25:34.974 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.986+0000 7f2e7ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:34.974 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.986+0000 7f2e957fa640 4 set_mon_vals no callback set 2024-03-29T11:25:34.974 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.987+0000 7f2e7ffff640 1 -- 172.21.15.104:0/2735132388 <== mon.2 v1:172.21.15.104:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f2e88001cd0 con 0x7f2e980a9ac0 2024-03-29T11:25:34.974 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.987+0000 7f2e7ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:34.974 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.987+0000 7f2e7ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:34.974 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.987+0000 7f2e7ffff640 10 monclient: dump: 2024-03-29T11:25:34.974 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:25:34.974 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:25:34.974 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:34.974 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:34.974 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:34.975 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:25:34.975 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:25:34.975 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:25:34.975 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:25:34.975 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:25:34.975 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.987+0000 7f2e9e0d9640 5 monclient: authenticate success, global_id 4466 2024-03-29T11:25:34.975 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.987+0000 7f2e9e0d9640 10 monclient: _renew_subs 2024-03-29T11:25:34.975 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.987+0000 7f2e9e0d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:25:34.975 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.987+0000 7f2e9e0d9640 1 -- 172.21.15.104:0/2735132388 --> v1:172.21.15.104:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e9814b490 con 0x7f2e980a9ac0 2024-03-29T11:25:34.975 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.987+0000 7f2e9e0d9640 10 client.4466.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:34.975 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.987+0000 7f2e9e0d9640 10 monclient: _renew_subs 2024-03-29T11:25:34.975 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.987+0000 7f2e9e0d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:25:34.975 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.987+0000 7f2e9e0d9640 1 -- 172.21.15.104:0/2735132388 --> v1:172.21.15.104:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e9812f650 con 0x7f2e980a9ac0 2024-03-29T11:25:34.975 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.987+0000 7f2e9e0d9640 1 librados: init done 2024-03-29T11:25:34.977 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.990+0000 7f2e7ffff640 1 -- 172.21.15.104:0/2735132388 <== mon.2 v1:172.21.15.104:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f2e8801ad40 con 0x7f2e980a9ac0 2024-03-29T11:25:34.977 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.991+0000 7f2e9e0d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:34.977 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.991+0000 7f2e9e0d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:34.977 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.991+0000 7f2e9e0d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:25:34.977 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.991+0000 7f2e7ffff640 1 -- 172.21.15.104:0/2735132388 <== mon.2 v1:172.21.15.104:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f2e88084b30 con 0x7f2e980a9ac0 2024-03-29T11:25:34.978 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.991+0000 7f2e7ffff640 10 client.4466.objecter ms_dispatch 0x7f2e98000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:34.978 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.991+0000 7f2e7ffff640 3 client.4466.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:34.978 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.991+0000 7f2e7ffff640 3 client.4466.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:34.978 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.991+0000 7f2e9e0d9640 1 -- 172.21.15.104:0/2735132388 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e9813fe00 con 0x7f2e980a9ac0 2024-03-29T11:25:34.978 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.991+0000 7f2e7ffff640 20 client.4466.objecter dump_active .. 0 homeless 2024-03-29T11:25:34.978 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.991+0000 7f2e96ffd640 10 monclient: get_auth_request con 0x7f2e8406dd00 auth_method 0 2024-03-29T11:25:34.978 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:34.992+0000 7f2e7ffff640 10 client.4466.objecter ms_handle_connect 0x7f2e8406dd00 2024-03-29T11:25:34.985 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:35.001+0000 7f2e7ffff640 1 -- 172.21.15.104:0/2735132388 <== mon.2 v1:172.21.15.104:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2e88050410 con 0x7f2e980a9ac0 2024-03-29T11:25:34.985 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:35.001+0000 7f2e7ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:34.985 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:35.001+0000 7f2e7ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:35.316 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:35.331+0000 7f2e9e0d9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:35.316 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:35.331+0000 7f2e9e0d9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:35.316 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:35.331+0000 7f2e9e0d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:25:35.316 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:35.331+0000 7f2e9e0d9640 1 -- 172.21.15.104:0/2735132388 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2e98075f10 con 0x7f2e980a9ac0 2024-03-29T11:25:35.317 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:35.332+0000 7f2e7ffff640 1 -- 172.21.15.104:0/2735132388 <== mon.2 v1:172.21.15.104:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f2e88052340 con 0x7f2e980a9ac0 2024-03-29T11:25:35.317 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:35.332+0000 7f2e7ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:35.317 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:35.332+0000 7f2e7ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:35.318 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:25:35.318 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:35.322 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:35.337+0000 7f2e9e0d9640 10 librados: watch_flush enter 2024-03-29T11:25:35.322 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:35.337+0000 7f2e9e0d9640 10 librados: watch_flush exit 2024-03-29T11:25:35.322 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:35.337+0000 7f2e9e0d9640 20 client.4466.objecter shutdown clearing up homeless session... 2024-03-29T11:25:35.322 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:35.337+0000 7f2e9e0d9640 10 client.4466.objecter successfully canceled tick 2024-03-29T11:25:35.322 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:35.337+0000 7f2e9e0d9640 1 -- 172.21.15.104:0/2735132388 >> v1:172.21.15.104:6816/3341126660 conn(0x7f2e8406dd00 legacy=0x7f2e840701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:35.322 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:35.337+0000 7f2e9e0d9640 10 monclient: shutdown 2024-03-29T11:25:35.322 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:35.337+0000 7f2e9e0d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:35.322 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:35.337+0000 7f2e9e0d9640 1 -- 172.21.15.104:0/2735132388 >> v1:172.21.15.104:6790/0 conn(0x7f2e980a9ac0 legacy=0x7f2e9812ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:35.322 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:35.337+0000 7f2e9e0d9640 1 -- 172.21.15.104:0/2735132388 shutdown_connections 2024-03-29T11:25:35.322 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:35.337+0000 7f2e9e0d9640 1 -- 172.21.15.104:0/2735132388 >> 172.21.15.104:0/2735132388 conn(0x7f2e980b35e0 msgr2=0x7f2e980adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:35.325 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:35.341+0000 7f2e9e0d9640 1 -- 172.21.15.104:0/2735132388 shutdown_connections 2024-03-29T11:25:35.325 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:35.341+0000 7f2e9e0d9640 1 -- 172.21.15.104:0/2735132388 wait complete. 2024-03-29T11:25:35.325 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:35.341+0000 7f2e9e0d9640 1 librados: shutdown 2024-03-29T11:25:35.345 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:42.153 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:42.306 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.321+0000 7eff3ee0e640 10 monclient: get_monmap_and_config 2024-03-29T11:25:42.306 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.321+0000 7eff3ee0e640 10 monclient: build_initial_monmap 2024-03-29T11:25:42.306 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.321+0000 7eff3ee0e640 10 monclient: monmap: 2024-03-29T11:25:42.306 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:25:42.306 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:25:42.306 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:25:42.322172+0000 2024-03-29T11:25:42.306 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:25:42.322172+0000 2024-03-29T11:25:42.306 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:42.307 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:25:42.307 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:25:42.307 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:25:42.307 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:25:42.307 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:25:42.308 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.322+0000 7eff3ee0e640 1 Processor -- start 2024-03-29T11:25:42.309 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.322+0000 7eff3ee0e640 1 -- start start 2024-03-29T11:25:42.309 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.322+0000 7eff3ee0e640 10 monclient: init 2024-03-29T11:25:42.309 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.323+0000 7eff3ee0e640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:42.309 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.323+0000 7eff3ee0e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:25:42.309 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.323+0000 7eff3ee0e640 10 monclient(hunting): picked mon.noname-c con 0x7eff381422b0 addr v1:172.21.15.119:6789/0 2024-03-29T11:25:42.309 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.323+0000 7eff3ee0e640 10 monclient(hunting): picked mon.noname-a con 0x7eff38142690 addr v1:172.21.15.104:6789/0 2024-03-29T11:25:42.309 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.323+0000 7eff3ee0e640 10 monclient(hunting): picked mon.noname-b con 0x7eff380b17d0 addr v1:172.21.15.104:6790/0 2024-03-29T11:25:42.309 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.323+0000 7eff3ee0e640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff38142c80 con 0x7eff38142690 2024-03-29T11:25:42.309 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.323+0000 7eff3ee0e640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff380aae40 con 0x7eff381422b0 2024-03-29T11:25:42.309 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.323+0000 7eff3ee0e640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff3813fbd0 con 0x7eff380b17d0 2024-03-29T11:25:42.309 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.323+0000 7eff3ee0e640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:42.309 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.323+0000 7eff3ee0e640 10 monclient(hunting): authenticate will time out at 1323.574458s 2024-03-29T11:25:42.311 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.326+0000 7eff3cbaa640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7eff381422b0 0x7eff3813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:50489/0 (socket says 172.21.15.104:50489) 2024-03-29T11:25:42.314 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.326+0000 7eff3cbaa640 1 -- 172.21.15.104:0/1651671829 learned_addr learned my addr 172.21.15.104:0/1651671829 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:25:42.319 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.330+0000 7eff377fe640 1 -- 172.21.15.104:0/1651671829 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3186557044 0 0) 0x7eff380aae40 con 0x7eff381422b0 2024-03-29T11:25:42.319 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.330+0000 7eff377fe640 10 monclient(hunting): _negotiate 2024-03-29T11:25:42.319 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.330+0000 7eff377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:42.319 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.330+0000 7eff377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:42.319 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.330+0000 7eff377fe640 10 monclient(hunting): my global_id is 4441 2024-03-29T11:25:42.320 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.330+0000 7eff377fe640 1 -- 172.21.15.104:0/1651671829 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff20003520 con 0x7eff381422b0 2024-03-29T11:25:42.320 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.330+0000 7eff377fe640 1 -- 172.21.15.104:0/1651671829 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 558002167 0 0) 0x7eff3813fbd0 con 0x7eff380b17d0 2024-03-29T11:25:42.320 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.330+0000 7eff377fe640 10 monclient(hunting): _negotiate 2024-03-29T11:25:42.320 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.330+0000 7eff377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:42.320 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.330+0000 7eff377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:42.320 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.330+0000 7eff377fe640 10 monclient(hunting): my global_id is 4469 2024-03-29T11:25:42.320 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.331+0000 7eff377fe640 1 -- 172.21.15.104:0/1651671829 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff380aae40 con 0x7eff380b17d0 2024-03-29T11:25:42.320 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.331+0000 7eff377fe640 1 -- 172.21.15.104:0/1651671829 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3816493225 0 0) 0x7eff20003520 con 0x7eff381422b0 2024-03-29T11:25:42.320 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.331+0000 7eff377fe640 1 -- 172.21.15.104:0/1651671829 >> v1:172.21.15.104:6790/0 conn(0x7eff380b17d0 legacy=0x7eff380b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:42.320 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.331+0000 7eff377fe640 1 -- 172.21.15.104:0/1651671829 >> v1:172.21.15.104:6789/0 conn(0x7eff38142690 legacy=0x7eff380b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:42.320 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.331+0000 7eff377fe640 10 monclient: _finish_hunting 0 2024-03-29T11:25:42.320 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.331+0000 7eff377fe640 1 monclient: found mon.noname-c 2024-03-29T11:25:42.320 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.331+0000 7eff377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:42.320 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.331+0000 7eff377fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.119:6789/0 2024-03-29T11:25:42.321 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.331+0000 7eff377fe640 1 -- 172.21.15.104:0/1651671829 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff38151940 con 0x7eff381422b0 2024-03-29T11:25:42.321 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.331+0000 7eff377fe640 10 monclient: _finish_auth 0 2024-03-29T11:25:42.321 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.331+0000 7eff377fe640 10 monclient: _check_auth_tickets 2024-03-29T11:25:42.321 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.331+0000 7eff377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:42.321 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.331+0000 7eff377fe640 1 -- 172.21.15.104:0/1651671829 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7eff30001a10 con 0x7eff381422b0 2024-03-29T11:25:42.321 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.332+0000 7eff377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:42.321 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.332+0000 7eff377fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:42.321 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.332+0000 7eff377fe640 10 monclient: dump: 2024-03-29T11:25:42.321 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:25:42.321 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:25:42.321 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:42.321 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:42.321 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:42.322 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:25:42.322 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:25:42.322 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:25:42.322 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:25:42.322 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:25:42.322 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.332+0000 7eff377fe640 1 -- 172.21.15.104:0/1651671829 <== mon.1 v1:172.21.15.119:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7eff30001c30 con 0x7eff381422b0 2024-03-29T11:25:42.322 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.332+0000 7eff377fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:42.322 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.332+0000 7eff377fe640 1 -- 172.21.15.104:0/1651671829 <== mon.1 v1:172.21.15.119:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7eff30001fc0 con 0x7eff381422b0 2024-03-29T11:25:42.322 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.332+0000 7eff377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:42.322 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.332+0000 7eff377fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:42.322 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.332+0000 7eff377fe640 10 monclient: dump: 2024-03-29T11:25:42.322 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:25:42.323 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:25:42.323 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:42.323 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:42.323 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:42.323 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:25:42.323 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:25:42.323 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:25:42.323 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:25:42.323 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:25:42.323 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.332+0000 7eff3ee0e640 5 monclient: authenticate success, global_id 4441 2024-03-29T11:25:42.323 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.332+0000 7eff3ee0e640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:42.323 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.332+0000 7eff3ee0e640 4 set_mon_vals no callback set 2024-03-29T11:25:42.323 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.333+0000 7eff3ee0e640 10 monclient: shutdown 2024-03-29T11:25:42.324 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.333+0000 7eff3ee0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:42.324 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.333+0000 7eff3ee0e640 1 -- 172.21.15.104:0/1651671829 >> v1:172.21.15.119:6789/0 conn(0x7eff381422b0 legacy=0x7eff3813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:42.324 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.334+0000 7eff3ee0e640 1 -- 172.21.15.104:0/1651671829 shutdown_connections 2024-03-29T11:25:42.324 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.334+0000 7eff3ee0e640 1 -- 172.21.15.104:0/1651671829 >> 172.21.15.104:0/1651671829 conn(0x7eff38066a00 msgr2=0x7eff38066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:42.324 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.335+0000 7eff3ee0e640 1 -- 172.21.15.104:0/1651671829 shutdown_connections 2024-03-29T11:25:42.324 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.335+0000 7eff3ee0e640 1 -- 172.21.15.104:0/1651671829 wait complete. 2024-03-29T11:25:42.325 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.336+0000 7eff3ee0e640 10 monclient: build_initial_monmap 2024-03-29T11:25:42.325 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.336+0000 7eff3ee0e640 10 monclient: monmap: 2024-03-29T11:25:42.325 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:25:42.325 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:42.325 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:25:42.325 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:25:42.325 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:42.325 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:25:42.325 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:25:42.326 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:25:42.326 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:25:42.326 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:25:42.326 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.336+0000 7eff3ee0e640 1 librados: starting msgr at 2024-03-29T11:25:42.326 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.336+0000 7eff3ee0e640 1 librados: starting objecter 2024-03-29T11:25:42.326 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.337+0000 7eff3ee0e640 1 Processor -- start 2024-03-29T11:25:42.326 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.337+0000 7eff3ee0e640 1 -- start start 2024-03-29T11:25:42.326 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.337+0000 7eff3ee0e640 1 librados: setting wanted keys 2024-03-29T11:25:42.326 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.337+0000 7eff3ee0e640 1 librados: calling monclient init 2024-03-29T11:25:42.326 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.337+0000 7eff3ee0e640 10 monclient: init 2024-03-29T11:25:42.327 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.338+0000 7eff3ee0e640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:42.327 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.338+0000 7eff3ee0e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:25:42.327 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.338+0000 7eff3ee0e640 10 monclient(hunting): picked mon.noname-a con 0x7eff380b17d0 addr v1:172.21.15.104:6789/0 2024-03-29T11:25:42.327 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.339+0000 7eff3ee0e640 10 monclient(hunting): picked mon.noname-b con 0x7eff381422b0 addr v1:172.21.15.119:6789/0 2024-03-29T11:25:42.327 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.342+0000 7eff37fff640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7eff381422b0 0x7eff380b0160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:50589/0 (socket says 172.21.15.104:50589) 2024-03-29T11:25:42.327 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.342+0000 7eff37fff640 1 -- 172.21.15.104:0/2101039792 learned_addr learned my addr 172.21.15.104:0/2101039792 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:25:42.327 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.343+0000 7eff3ee0e640 10 monclient(hunting): picked mon.noname-c con 0x7eff38142690 addr v1:172.21.15.104:6790/0 2024-03-29T11:25:42.328 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.343+0000 7eff3ee0e640 1 -- 172.21.15.104:0/2101039792 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff381db9e0 con 0x7eff380b17d0 2024-03-29T11:25:42.328 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.343+0000 7eff357fa640 10 client.?.objecter ms_handle_connect 0x7eff381422b0 2024-03-29T11:25:42.328 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.343+0000 7eff357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:42.328 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.343+0000 7eff357fa640 10 client.?.objecter ms_handle_connect 0x7eff380b17d0 2024-03-29T11:25:42.328 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.343+0000 7eff357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:42.328 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.343+0000 7eff3ee0e640 1 -- 172.21.15.104:0/2101039792 --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff381dbb70 con 0x7eff381422b0 2024-03-29T11:25:42.328 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.343+0000 7eff3ee0e640 1 -- 172.21.15.104:0/2101039792 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff380ad0e0 con 0x7eff38142690 2024-03-29T11:25:42.328 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.343+0000 7eff3ee0e640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:42.328 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.343+0000 7eff3ee0e640 10 monclient(hunting): authenticate will time out at 1323.594331s 2024-03-29T11:25:42.330 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.344+0000 7eff357fa640 10 client.?.objecter ms_handle_connect 0x7eff38142690 2024-03-29T11:25:42.330 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.344+0000 7eff357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:42.330 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.344+0000 7eff357fa640 1 -- 172.21.15.104:0/2101039792 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 370542810 0 0) 0x7eff380ad0e0 con 0x7eff38142690 2024-03-29T11:25:42.330 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.344+0000 7eff357fa640 10 monclient(hunting): _negotiate 2024-03-29T11:25:42.331 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.344+0000 7eff357fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:42.331 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.344+0000 7eff357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:42.331 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.344+0000 7eff357fa640 10 monclient(hunting): my global_id is 4472 2024-03-29T11:25:42.331 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.344+0000 7eff357fa640 1 -- 172.21.15.104:0/2101039792 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff24004eb0 con 0x7eff38142690 2024-03-29T11:25:42.331 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.344+0000 7eff357fa640 1 -- 172.21.15.104:0/2101039792 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3937552196 0 0) 0x7eff24004eb0 con 0x7eff38142690 2024-03-29T11:25:42.331 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.344+0000 7eff357fa640 1 -- 172.21.15.104:0/2101039792 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7eff380ad0e0 con 0x7eff38142690 2024-03-29T11:25:42.331 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.344+0000 7eff357fa640 1 -- 172.21.15.104:0/2101039792 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7eff28001930 con 0x7eff38142690 2024-03-29T11:25:42.331 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.344+0000 7eff357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:42.331 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.344+0000 7eff357fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:42.331 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.344+0000 7eff357fa640 10 monclient(hunting): dump: 2024-03-29T11:25:42.331 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:25:42.331 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:25:42.332 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:42.332 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:42.332 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:42.332 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:25:42.332 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:25:42.332 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:25:42.332 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:25:42.332 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:25:42.332 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.344+0000 7eff357fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:42.332 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.345+0000 7eff357fa640 1 -- 172.21.15.104:0/2101039792 <== mon.2 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1446771626 0 0) 0x7eff380ad0e0 con 0x7eff38142690 2024-03-29T11:25:42.332 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.345+0000 7eff357fa640 1 -- 172.21.15.104:0/2101039792 >> v1:172.21.15.119:6789/0 conn(0x7eff381422b0 legacy=0x7eff380b0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:42.332 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.345+0000 7eff357fa640 1 -- 172.21.15.104:0/2101039792 >> v1:172.21.15.104:6789/0 conn(0x7eff380b17d0 legacy=0x7eff381daf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:42.332 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.345+0000 7eff357fa640 10 monclient: _finish_hunting 0 2024-03-29T11:25:42.332 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.345+0000 7eff357fa640 1 monclient: found mon.c 2024-03-29T11:25:42.332 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.345+0000 7eff357fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:42.333 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.345+0000 7eff357fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:25:42.333 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.345+0000 7eff357fa640 1 -- 172.21.15.104:0/2101039792 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff380ad270 con 0x7eff38142690 2024-03-29T11:25:42.333 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.345+0000 7eff357fa640 10 monclient: _finish_auth 0 2024-03-29T11:25:42.333 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.345+0000 7eff357fa640 10 monclient: _check_auth_tickets 2024-03-29T11:25:42.333 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.345+0000 7eff357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:42.333 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.345+0000 7eff3ee0e640 5 monclient: authenticate success, global_id 4472 2024-03-29T11:25:42.333 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.345+0000 7eff3ee0e640 10 monclient: _renew_subs 2024-03-29T11:25:42.333 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.345+0000 7eff3ee0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:25:42.333 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.345+0000 7eff3ee0e640 1 -- 172.21.15.104:0/2101039792 --> v1:172.21.15.104:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff3814daf0 con 0x7eff38142690 2024-03-29T11:25:42.333 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.345+0000 7eff3ee0e640 10 client.4472.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:42.333 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.345+0000 7eff3ee0e640 10 monclient: _renew_subs 2024-03-29T11:25:42.333 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.345+0000 7eff3ee0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:25:42.333 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.345+0000 7eff3ee0e640 1 -- 172.21.15.104:0/2101039792 --> v1:172.21.15.104:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7eff3814de40 con 0x7eff38142690 2024-03-29T11:25:42.333 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.345+0000 7eff3ee0e640 1 librados: init done 2024-03-29T11:25:42.334 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.347+0000 7eff357fa640 1 -- 172.21.15.104:0/2101039792 <== mon.2 v1:172.21.15.104:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7eff28003610 con 0x7eff38142690 2024-03-29T11:25:42.334 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.347+0000 7eff357fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:42.334 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.347+0000 7eff357fa640 1 -- 172.21.15.104:0/2101039792 <== mon.2 v1:172.21.15.104:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7eff28003920 con 0x7eff38142690 2024-03-29T11:25:42.334 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.347+0000 7eff357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:42.334 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.347+0000 7eff357fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:42.335 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.347+0000 7eff357fa640 10 monclient: dump: 2024-03-29T11:25:42.335 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:25:42.335 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:25:42.335 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:42.335 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:42.335 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:42.335 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:25:42.335 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:25:42.335 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:25:42.335 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:25:42.335 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:25:42.335 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.347+0000 7eff36ffd640 4 set_mon_vals no callback set 2024-03-29T11:25:42.336 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.348+0000 7eff3ee0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:42.336 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.348+0000 7eff357fa640 1 -- 172.21.15.104:0/2101039792 <== mon.2 v1:172.21.15.104:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7eff280198f0 con 0x7eff38142690 2024-03-29T11:25:42.336 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.349+0000 7eff3ee0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:42.336 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.349+0000 7eff3ee0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:25:42.336 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.349+0000 7eff3ee0e640 1 -- 172.21.15.104:0/2101039792 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff18005080 con 0x7eff38142690 2024-03-29T11:25:42.336 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.349+0000 7eff357fa640 1 -- 172.21.15.104:0/2101039792 <== mon.2 v1:172.21.15.104:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7eff28083850 con 0x7eff38142690 2024-03-29T11:25:42.336 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.349+0000 7eff357fa640 10 client.4472.objecter ms_dispatch 0x7eff38000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:42.336 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.349+0000 7eff357fa640 3 client.4472.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:42.336 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.349+0000 7eff357fa640 3 client.4472.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:42.336 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.349+0000 7eff357fa640 20 client.4472.objecter dump_active .. 0 homeless 2024-03-29T11:25:42.337 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.349+0000 7eff3cbaa640 10 monclient: get_auth_request con 0x7eff2406dcd0 auth_method 0 2024-03-29T11:25:42.337 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.349+0000 7eff357fa640 10 client.4472.objecter ms_handle_connect 0x7eff2406dcd0 2024-03-29T11:25:42.345 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.360+0000 7eff357fa640 1 -- 172.21.15.104:0/2101039792 <== mon.2 v1:172.21.15.104:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7eff2804f130 con 0x7eff38142690 2024-03-29T11:25:42.345 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.360+0000 7eff357fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:42.345 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.360+0000 7eff357fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:42.675 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.690+0000 7eff3ee0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:42.675 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.690+0000 7eff3ee0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:42.675 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.690+0000 7eff3ee0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:25:42.675 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.690+0000 7eff3ee0e640 1 -- 172.21.15.104:0/2101039792 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7eff18005330 con 0x7eff38142690 2024-03-29T11:25:42.676 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.692+0000 7eff357fa640 1 -- 172.21.15.104:0/2101039792 <== mon.2 v1:172.21.15.104:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7eff28051060 con 0x7eff38142690 2024-03-29T11:25:42.676 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.692+0000 7eff357fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:42.676 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.692+0000 7eff357fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:42.678 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:25:42.678 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:42.681 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.695+0000 7eff3ee0e640 10 librados: watch_flush enter 2024-03-29T11:25:42.681 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.695+0000 7eff3ee0e640 10 librados: watch_flush exit 2024-03-29T11:25:42.681 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.695+0000 7eff3ee0e640 20 client.4472.objecter shutdown clearing up homeless session... 2024-03-29T11:25:42.681 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.695+0000 7eff3ee0e640 10 client.4472.objecter successfully canceled tick 2024-03-29T11:25:42.681 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.695+0000 7eff3ee0e640 1 -- 172.21.15.104:0/2101039792 >> v1:172.21.15.104:6816/3341126660 conn(0x7eff2406dcd0 legacy=0x7eff24070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:42.681 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.695+0000 7eff3ee0e640 10 monclient: shutdown 2024-03-29T11:25:42.681 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.695+0000 7eff3ee0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:42.681 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.695+0000 7eff3ee0e640 1 -- 172.21.15.104:0/2101039792 >> v1:172.21.15.104:6790/0 conn(0x7eff38142690 legacy=0x7eff380ac9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:42.681 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.696+0000 7eff3ee0e640 1 -- 172.21.15.104:0/2101039792 shutdown_connections 2024-03-29T11:25:42.681 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.696+0000 7eff3ee0e640 1 -- 172.21.15.104:0/2101039792 >> 172.21.15.104:0/2101039792 conn(0x7eff38066a00 msgr2=0x7eff380b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:42.681 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.696+0000 7eff3ee0e640 1 -- 172.21.15.104:0/2101039792 shutdown_connections 2024-03-29T11:25:42.681 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.696+0000 7eff3ee0e640 1 -- 172.21.15.104:0/2101039792 wait complete. 2024-03-29T11:25:42.682 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:42.696+0000 7eff3ee0e640 1 librados: shutdown 2024-03-29T11:25:42.699 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:49.506 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:49.639 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.654+0000 7f5f306c5640 10 monclient: get_monmap_and_config 2024-03-29T11:25:49.639 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.654+0000 7f5f306c5640 10 monclient: build_initial_monmap 2024-03-29T11:25:49.639 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.654+0000 7f5f306c5640 10 monclient: monmap: 2024-03-29T11:25:49.639 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:25:49.640 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:25:49.640 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:25:49.655592+0000 2024-03-29T11:25:49.640 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:25:49.655592+0000 2024-03-29T11:25:49.640 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:49.640 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:25:49.640 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:25:49.640 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:25:49.640 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:25:49.640 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:25:49.641 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.655+0000 7f5f306c5640 1 Processor -- start 2024-03-29T11:25:49.641 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.655+0000 7f5f306c5640 1 -- start start 2024-03-29T11:25:49.641 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.655+0000 7f5f306c5640 10 monclient: init 2024-03-29T11:25:49.643 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.657+0000 7f5f306c5640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:49.643 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.657+0000 7f5f306c5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:25:49.643 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.657+0000 7f5f306c5640 10 monclient(hunting): picked mon.noname-a con 0x7f5f280b4790 addr v1:172.21.15.104:6789/0 2024-03-29T11:25:49.643 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.657+0000 7f5f306c5640 10 monclient(hunting): picked mon.noname-b con 0x7f5f280aea90 addr v1:172.21.15.104:6790/0 2024-03-29T11:25:49.643 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.657+0000 7f5f2dc60640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f5f280aea90 0x7f5f280af540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:43948/0 (socket says 172.21.15.104:43948) 2024-03-29T11:25:49.643 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.657+0000 7f5f2dc60640 1 -- 172.21.15.104:0/4087943091 learned_addr learned my addr 172.21.15.104:0/4087943091 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:25:49.643 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.657+0000 7f5f306c5640 10 monclient(hunting): picked mon.noname-c con 0x7f5f280b39f0 addr v1:172.21.15.119:6789/0 2024-03-29T11:25:49.643 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.657+0000 7f5f306c5640 1 -- 172.21.15.104:0/4087943091 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f280b0c60 con 0x7f5f280b4790 2024-03-29T11:25:49.644 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.657+0000 7f5f306c5640 1 -- 172.21.15.104:0/4087943091 --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f280af020 con 0x7f5f280b39f0 2024-03-29T11:25:49.644 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.657+0000 7f5f306c5640 1 -- 172.21.15.104:0/4087943091 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f2813fbd0 con 0x7f5f280aea90 2024-03-29T11:25:49.644 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.657+0000 7f5f306c5640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:49.644 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.657+0000 7f5f306c5640 10 monclient(hunting): authenticate will time out at 1330.908846s 2024-03-29T11:25:49.644 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.658+0000 7f5f2d45f640 1 -- 172.21.15.104:0/4087943091 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1210212728 0 0) 0x7f5f2813fbd0 con 0x7f5f280aea90 2024-03-29T11:25:49.644 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.658+0000 7f5f2d45f640 10 monclient(hunting): _negotiate 2024-03-29T11:25:49.644 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.658+0000 7f5f2d45f640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:49.644 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.658+0000 7f5f2d45f640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:49.644 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.658+0000 7f5f2d45f640 10 monclient(hunting): my global_id is 4475 2024-03-29T11:25:49.644 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.658+0000 7f5f2d45f640 1 -- 172.21.15.104:0/4087943091 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f14003520 con 0x7f5f280aea90 2024-03-29T11:25:49.644 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.658+0000 7f5f2d45f640 1 -- 172.21.15.104:0/4087943091 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2755686110 0 0) 0x7f5f14003520 con 0x7f5f280aea90 2024-03-29T11:25:49.644 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.658+0000 7f5f2d45f640 1 -- 172.21.15.104:0/4087943091 >> v1:172.21.15.119:6789/0 conn(0x7f5f280b39f0 legacy=0x7f5f2813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:25:49.644 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.658+0000 7f5f2d45f640 1 -- 172.21.15.104:0/4087943091 >> v1:172.21.15.104:6789/0 conn(0x7f5f280b4790 legacy=0x7f5f280b1080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:49.646 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.661+0000 7f5f2d45f640 10 monclient: _finish_hunting 0 2024-03-29T11:25:49.646 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.661+0000 7f5f2d45f640 1 monclient: found mon.noname-b 2024-03-29T11:25:49.646 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.661+0000 7f5f2d45f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:49.646 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.661+0000 7f5f2d45f640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.104:6790/0 2024-03-29T11:25:49.646 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.661+0000 7f5f2d45f640 1 -- 172.21.15.104:0/4087943091 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f2813fd60 con 0x7f5f280aea90 2024-03-29T11:25:49.647 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.661+0000 7f5f2d45f640 10 monclient: _finish_auth 0 2024-03-29T11:25:49.647 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.661+0000 7f5f2d45f640 10 monclient: _check_auth_tickets 2024-03-29T11:25:49.647 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.661+0000 7f5f2d45f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:49.647 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.661+0000 7f5f2d45f640 1 -- 172.21.15.104:0/4087943091 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f5f24001670 con 0x7f5f280aea90 2024-03-29T11:25:49.647 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.661+0000 7f5f2d45f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:49.647 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.661+0000 7f5f2d45f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:49.647 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.661+0000 7f5f2d45f640 10 monclient: dump: 2024-03-29T11:25:49.647 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:25:49.647 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:25:49.647 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:49.648 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:49.648 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:49.648 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:25:49.648 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:25:49.648 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:25:49.648 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:25:49.648 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:25:49.648 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.661+0000 7f5f306c5640 5 monclient: authenticate success, global_id 4475 2024-03-29T11:25:49.648 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.661+0000 7f5f306c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:49.648 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.661+0000 7f5f2d45f640 1 -- 172.21.15.104:0/4087943091 <== mon.2 v1:172.21.15.104:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f240018d0 con 0x7f5f280aea90 2024-03-29T11:25:49.648 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.661+0000 7f5f2d45f640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:49.648 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.661+0000 7f5f2d45f640 1 -- 172.21.15.104:0/4087943091 <== mon.2 v1:172.21.15.104:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f5f24001d20 con 0x7f5f280aea90 2024-03-29T11:25:49.648 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.661+0000 7f5f2d45f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:49.648 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.661+0000 7f5f2d45f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:49.649 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.661+0000 7f5f2d45f640 10 monclient: dump: 2024-03-29T11:25:49.649 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:25:49.649 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:25:49.649 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:49.649 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:49.649 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:49.649 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:25:49.649 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:25:49.649 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:25:49.649 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:25:49.649 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:25:49.649 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.661+0000 7f5f306c5640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:49.649 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.661+0000 7f5f306c5640 4 set_mon_vals no callback set 2024-03-29T11:25:49.654 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.662+0000 7f5f306c5640 10 monclient: shutdown 2024-03-29T11:25:49.654 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.662+0000 7f5f306c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:49.654 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.662+0000 7f5f306c5640 1 -- 172.21.15.104:0/4087943091 >> v1:172.21.15.104:6790/0 conn(0x7f5f280aea90 legacy=0x7f5f280af540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:49.654 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.663+0000 7f5f306c5640 1 -- 172.21.15.104:0/4087943091 shutdown_connections 2024-03-29T11:25:49.654 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.663+0000 7f5f306c5640 1 -- 172.21.15.104:0/4087943091 >> 172.21.15.104:0/4087943091 conn(0x7f5f28066a00 msgr2=0x7f5f28066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:49.654 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.663+0000 7f5f306c5640 1 -- 172.21.15.104:0/4087943091 shutdown_connections 2024-03-29T11:25:49.654 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.663+0000 7f5f306c5640 1 -- 172.21.15.104:0/4087943091 wait complete. 2024-03-29T11:25:49.654 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.664+0000 7f5f306c5640 10 monclient: build_initial_monmap 2024-03-29T11:25:49.655 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.664+0000 7f5f306c5640 10 monclient: monmap: 2024-03-29T11:25:49.655 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:25:49.655 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:49.655 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:25:49.655 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:25:49.655 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:49.655 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:25:49.655 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:25:49.655 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:25:49.655 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:25:49.655 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:25:49.655 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.664+0000 7f5f306c5640 1 librados: starting msgr at 2024-03-29T11:25:49.656 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.664+0000 7f5f306c5640 1 librados: starting objecter 2024-03-29T11:25:49.656 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.664+0000 7f5f306c5640 1 Processor -- start 2024-03-29T11:25:49.656 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.664+0000 7f5f306c5640 1 -- start start 2024-03-29T11:25:49.656 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.664+0000 7f5f306c5640 1 librados: setting wanted keys 2024-03-29T11:25:49.656 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.664+0000 7f5f306c5640 1 librados: calling monclient init 2024-03-29T11:25:49.656 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.664+0000 7f5f306c5640 10 monclient: init 2024-03-29T11:25:49.656 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.665+0000 7f5f306c5640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:49.656 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.665+0000 7f5f306c5640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:25:49.656 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.665+0000 7f5f306c5640 10 monclient(hunting): picked mon.noname-a con 0x7f5f280b39f0 addr v1:172.21.15.104:6789/0 2024-03-29T11:25:49.656 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.665+0000 7f5f306c5640 10 monclient(hunting): picked mon.noname-b con 0x7f5f280b4790 addr v1:172.21.15.119:6789/0 2024-03-29T11:25:49.656 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.665+0000 7f5f306c5640 10 monclient(hunting): picked mon.noname-c con 0x7f5f28070f10 addr v1:172.21.15.104:6790/0 2024-03-29T11:25:49.656 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.665+0000 7f5f306c5640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f28134a70 con 0x7f5f280b39f0 2024-03-29T11:25:49.656 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.665+0000 7f5f306c5640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f28071430 con 0x7f5f280b4790 2024-03-29T11:25:49.657 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.665+0000 7f5f306c5640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f281347f0 con 0x7f5f28070f10 2024-03-29T11:25:49.657 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.665+0000 7f5f306c5640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:49.657 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.665+0000 7f5f306c5640 10 monclient(hunting): authenticate will time out at 1330.916216s 2024-03-29T11:25:49.657 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.665+0000 7f5f2dc60640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f5f280b4790 0x7f5f281339d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:39054/0 (socket says 172.21.15.104:39054) 2024-03-29T11:25:49.657 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.665+0000 7f5f2dc60640 1 -- 172.21.15.104:0/216418649 learned_addr learned my addr 172.21.15.104:0/216418649 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:25:49.657 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.665+0000 7f5f1effd640 10 client.?.objecter ms_handle_connect 0x7f5f280b4790 2024-03-29T11:25:49.657 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.665+0000 7f5f1effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:49.657 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.665+0000 7f5f1effd640 10 client.?.objecter ms_handle_connect 0x7f5f28070f10 2024-03-29T11:25:49.657 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.665+0000 7f5f1effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:49.657 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.666+0000 7f5f1effd640 1 -- 172.21.15.104:0/216418649 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3102776163 0 0) 0x7f5f281347f0 con 0x7f5f28070f10 2024-03-29T11:25:49.657 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.666+0000 7f5f1effd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:49.657 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.666+0000 7f5f1effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:49.657 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.666+0000 7f5f1effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:49.657 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.666+0000 7f5f1effd640 10 monclient(hunting): my global_id is 4478 2024-03-29T11:25:49.657 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.666+0000 7f5f1effd640 1 -- 172.21.15.104:0/216418649 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f0c004eb0 con 0x7f5f28070f10 2024-03-29T11:25:49.658 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.667+0000 7f5f1effd640 1 -- 172.21.15.104:0/216418649 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 34949890 0 0) 0x7f5f0c004eb0 con 0x7f5f28070f10 2024-03-29T11:25:49.658 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.667+0000 7f5f1effd640 1 -- 172.21.15.104:0/216418649 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f281347f0 con 0x7f5f28070f10 2024-03-29T11:25:49.658 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.667+0000 7f5f1effd640 1 -- 172.21.15.104:0/216418649 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f5f20001890 con 0x7f5f28070f10 2024-03-29T11:25:49.658 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.667+0000 7f5f1effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:49.658 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.667+0000 7f5f1effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:49.658 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.667+0000 7f5f1effd640 10 monclient(hunting): dump: 2024-03-29T11:25:49.658 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:25:49.658 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:25:49.658 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:49.658 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:49.658 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:49.658 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:25:49.658 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:25:49.659 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:25:49.659 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:25:49.659 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:25:49.659 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.667+0000 7f5f1effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:49.659 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.667+0000 7f5f1effd640 1 -- 172.21.15.104:0/216418649 <== mon.2 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1143368581 0 0) 0x7f5f281347f0 con 0x7f5f28070f10 2024-03-29T11:25:49.659 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.667+0000 7f5f1effd640 1 -- 172.21.15.104:0/216418649 >> v1:172.21.15.119:6789/0 conn(0x7f5f280b4790 legacy=0x7f5f281339d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:49.659 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.667+0000 7f5f1effd640 1 -- 172.21.15.104:0/216418649 >> v1:172.21.15.104:6789/0 conn(0x7f5f280b39f0 legacy=0x7f5f2806de90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:25:49.659 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.668+0000 7f5f1effd640 10 monclient: _finish_hunting 0 2024-03-29T11:25:49.659 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.668+0000 7f5f1effd640 1 monclient: found mon.c 2024-03-29T11:25:49.659 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.668+0000 7f5f1effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:49.659 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.668+0000 7f5f1effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:25:49.659 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.668+0000 7f5f1effd640 1 -- 172.21.15.104:0/216418649 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f280a9860 con 0x7f5f28070f10 2024-03-29T11:25:49.659 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.668+0000 7f5f1effd640 10 monclient: _finish_auth 0 2024-03-29T11:25:49.660 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.668+0000 7f5f1effd640 10 monclient: _check_auth_tickets 2024-03-29T11:25:49.660 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.668+0000 7f5f1effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:49.660 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.668+0000 7f5f306c5640 5 monclient: authenticate success, global_id 4478 2024-03-29T11:25:49.660 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.668+0000 7f5f306c5640 10 monclient: _renew_subs 2024-03-29T11:25:49.660 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.668+0000 7f5f306c5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:25:49.660 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.668+0000 7f5f306c5640 1 -- 172.21.15.104:0/216418649 --> v1:172.21.15.104:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f280a9b40 con 0x7f5f28070f10 2024-03-29T11:25:49.660 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.668+0000 7f5f306c5640 10 client.4478.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:49.660 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.668+0000 7f5f306c5640 10 monclient: _renew_subs 2024-03-29T11:25:49.660 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.668+0000 7f5f306c5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:25:49.660 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.668+0000 7f5f306c5640 1 -- 172.21.15.104:0/216418649 --> v1:172.21.15.104:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f28071aa0 con 0x7f5f28070f10 2024-03-29T11:25:49.660 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.668+0000 7f5f306c5640 1 librados: init done 2024-03-29T11:25:49.660 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.668+0000 7f5f1effd640 1 -- 172.21.15.104:0/216418649 <== mon.2 v1:172.21.15.104:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f20001e20 con 0x7f5f28070f10 2024-03-29T11:25:49.660 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.668+0000 7f5f1effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:49.661 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.668+0000 7f5f1effd640 1 -- 172.21.15.104:0/216418649 <== mon.2 v1:172.21.15.104:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f5f20003c10 con 0x7f5f28070f10 2024-03-29T11:25:49.661 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.668+0000 7f5f1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:49.661 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.668+0000 7f5f1effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:49.661 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.668+0000 7f5f1effd640 10 monclient: dump: 2024-03-29T11:25:49.661 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:25:49.661 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:25:49.661 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:49.661 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:49.661 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:49.661 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:25:49.661 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:25:49.662 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:25:49.662 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:25:49.662 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:25:49.662 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.668+0000 7f5f2cc5e640 4 set_mon_vals no callback set 2024-03-29T11:25:49.663 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.670+0000 7f5f1effd640 1 -- 172.21.15.104:0/216418649 <== mon.2 v1:172.21.15.104:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f5f20019be0 con 0x7f5f28070f10 2024-03-29T11:25:49.663 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.671+0000 7f5f306c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:49.663 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.671+0000 7f5f306c5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:49.663 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.671+0000 7f5f306c5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:25:49.663 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.671+0000 7f5f1effd640 1 -- 172.21.15.104:0/216418649 <== mon.2 v1:172.21.15.104:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f5f20083bf0 con 0x7f5f28070f10 2024-03-29T11:25:49.664 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.671+0000 7f5f1effd640 10 client.4478.objecter ms_dispatch 0x7f5f28000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:49.664 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.671+0000 7f5f1effd640 3 client.4478.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:49.664 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.671+0000 7f5f1effd640 3 client.4478.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:49.664 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.671+0000 7f5f1effd640 20 client.4478.objecter dump_active .. 0 homeless 2024-03-29T11:25:49.664 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.671+0000 7f5f306c5640 1 -- 172.21.15.104:0/216418649 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f10005080 con 0x7f5f28070f10 2024-03-29T11:25:49.664 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.671+0000 7f5f2e461640 10 monclient: get_auth_request con 0x7f5f0c06d9c0 auth_method 0 2024-03-29T11:25:49.664 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.671+0000 7f5f1effd640 10 client.4478.objecter ms_handle_connect 0x7f5f0c06d9c0 2024-03-29T11:25:49.665 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.679+0000 7f5f1effd640 1 -- 172.21.15.104:0/216418649 <== mon.2 v1:172.21.15.104:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5f2004f300 con 0x7f5f28070f10 2024-03-29T11:25:49.665 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.679+0000 7f5f1effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:49.665 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.679+0000 7f5f1effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:49.970 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.984+0000 7f5f306c5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:49.970 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.984+0000 7f5f306c5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:49.970 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.984+0000 7f5f306c5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:25:49.970 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.984+0000 7f5f306c5640 1 -- 172.21.15.104:0/216418649 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f10005330 con 0x7f5f28070f10 2024-03-29T11:25:49.970 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.985+0000 7f5f1effd640 1 -- 172.21.15.104:0/216418649 <== mon.2 v1:172.21.15.104:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f5f20051400 con 0x7f5f28070f10 2024-03-29T11:25:49.970 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.985+0000 7f5f1effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:49.970 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.985+0000 7f5f1effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:49.970 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:25:49.971 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:49.973 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.988+0000 7f5f306c5640 10 librados: watch_flush enter 2024-03-29T11:25:49.973 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.988+0000 7f5f306c5640 10 librados: watch_flush exit 2024-03-29T11:25:49.973 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.988+0000 7f5f306c5640 20 client.4478.objecter shutdown clearing up homeless session... 2024-03-29T11:25:49.973 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.988+0000 7f5f306c5640 10 client.4478.objecter successfully canceled tick 2024-03-29T11:25:49.973 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.988+0000 7f5f306c5640 1 -- 172.21.15.104:0/216418649 >> v1:172.21.15.104:6816/3341126660 conn(0x7f5f0c06d9c0 legacy=0x7f5f0c06fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:49.973 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.988+0000 7f5f306c5640 10 monclient: shutdown 2024-03-29T11:25:49.973 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.988+0000 7f5f306c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:49.973 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.988+0000 7f5f306c5640 1 -- 172.21.15.104:0/216418649 >> v1:172.21.15.104:6790/0 conn(0x7f5f28070f10 legacy=0x7f5f281340e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:49.974 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.989+0000 7f5f306c5640 1 -- 172.21.15.104:0/216418649 shutdown_connections 2024-03-29T11:25:49.974 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.989+0000 7f5f306c5640 1 -- 172.21.15.104:0/216418649 >> 172.21.15.104:0/216418649 conn(0x7f5f28066a00 msgr2=0x7f5f280b1080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:49.974 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.989+0000 7f5f306c5640 1 -- 172.21.15.104:0/216418649 shutdown_connections 2024-03-29T11:25:49.974 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.989+0000 7f5f306c5640 1 -- 172.21.15.104:0/216418649 wait complete. 2024-03-29T11:25:49.974 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:49.989+0000 7f5f306c5640 1 librados: shutdown 2024-03-29T11:25:49.987 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:56.794 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:56.959 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.974+0000 7f92ed8dd640 10 monclient: get_monmap_and_config 2024-03-29T11:25:56.960 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.974+0000 7f92ed8dd640 10 monclient: build_initial_monmap 2024-03-29T11:25:56.960 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.974+0000 7f92ed8dd640 10 monclient: monmap: 2024-03-29T11:25:56.960 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:25:56.960 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:25:56.960 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:25:56.975718+0000 2024-03-29T11:25:56.960 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:25:56.975718+0000 2024-03-29T11:25:56.960 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:56.960 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:25:56.960 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:25:56.960 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:25:56.961 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:25:56.961 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:25:56.961 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.976+0000 7f92ed8dd640 1 Processor -- start 2024-03-29T11:25:56.961 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.976+0000 7f92ed8dd640 1 -- start start 2024-03-29T11:25:56.961 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.976+0000 7f92ed8dd640 10 monclient: init 2024-03-29T11:25:56.963 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.977+0000 7f92ed8dd640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:56.963 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.977+0000 7f92ed8dd640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:25:56.963 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.977+0000 7f92ed8dd640 10 monclient(hunting): picked mon.noname-a con 0x7f92e80b1f60 addr v1:172.21.15.104:6789/0 2024-03-29T11:25:56.963 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.977+0000 7f92ed8dd640 10 monclient(hunting): picked mon.noname-c con 0x7f92e8071eb0 addr v1:172.21.15.119:6789/0 2024-03-29T11:25:56.963 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.977+0000 7f92ed8dd640 10 monclient(hunting): picked mon.noname-b con 0x7f92e8143d10 addr v1:172.21.15.104:6790/0 2024-03-29T11:25:56.964 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.977+0000 7f92ed8dd640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92e806ef80 con 0x7f92e80b1f60 2024-03-29T11:25:56.964 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.977+0000 7f92ed8dd640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92e806f260 con 0x7f92e8071eb0 2024-03-29T11:25:56.964 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.977+0000 7f92ed8dd640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92e80b40c0 con 0x7f92e8143d10 2024-03-29T11:25:56.964 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.977+0000 7f92ed8dd640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:56.964 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.977+0000 7f92ed8dd640 10 monclient(hunting): authenticate will time out at 1338.228542s 2024-03-29T11:25:56.964 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.977+0000 7f92e67fc640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f92e8071eb0 0x7f92e8146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:37390/0 (socket says 172.21.15.104:37390) 2024-03-29T11:25:56.964 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.977+0000 7f92e67fc640 1 -- 172.21.15.104:0/2476327651 learned_addr learned my addr 172.21.15.104:0/2476327651 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:25:56.964 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.978+0000 7f92e5ffb640 1 -- 172.21.15.104:0/2476327651 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3341567298 0 0) 0x7f92e806f260 con 0x7f92e8071eb0 2024-03-29T11:25:56.964 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.978+0000 7f92e5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:56.964 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.978+0000 7f92e5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:56.964 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.978+0000 7f92e5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:56.964 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.978+0000 7f92e5ffb640 10 monclient(hunting): my global_id is 4450 2024-03-29T11:25:56.964 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.978+0000 7f92e5ffb640 1 -- 172.21.15.104:0/2476327651 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92d0003520 con 0x7f92e8071eb0 2024-03-29T11:25:56.966 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.979+0000 7f92e5ffb640 1 -- 172.21.15.104:0/2476327651 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1643434038 0 0) 0x7f92d0003520 con 0x7f92e8071eb0 2024-03-29T11:25:56.966 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.979+0000 7f92e5ffb640 1 -- 172.21.15.104:0/2476327651 >> v1:172.21.15.104:6790/0 conn(0x7f92e8143d10 legacy=0x7f92e8146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:56.966 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.979+0000 7f92e5ffb640 1 -- 172.21.15.104:0/2476327651 >> v1:172.21.15.104:6789/0 conn(0x7f92e80b1f60 legacy=0x7f92e8143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:56.966 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.979+0000 7f92e5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:25:56.966 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.979+0000 7f92e5ffb640 1 monclient: found mon.noname-c 2024-03-29T11:25:56.966 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.979+0000 7f92e5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:56.966 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.979+0000 7f92e5ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.119:6789/0 2024-03-29T11:25:56.966 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.979+0000 7f92e5ffb640 1 -- 172.21.15.104:0/2476327651 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92e8155d70 con 0x7f92e8071eb0 2024-03-29T11:25:56.966 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.979+0000 7f92e5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:25:56.966 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.979+0000 7f92e5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:25:56.967 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.979+0000 7f92e5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:56.967 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.979+0000 7f92e5ffb640 1 -- 172.21.15.104:0/2476327651 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f92dc001690 con 0x7f92e8071eb0 2024-03-29T11:25:56.967 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.979+0000 7f92e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:56.967 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.979+0000 7f92e5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:56.967 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.979+0000 7f92e5ffb640 10 monclient: dump: 2024-03-29T11:25:56.967 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:25:56.967 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:25:56.967 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:56.967 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:56.967 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:56.967 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:25:56.967 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:25:56.967 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:25:56.967 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:25:56.968 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:25:56.968 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.979+0000 7f92ed8dd640 5 monclient: authenticate success, global_id 4450 2024-03-29T11:25:56.968 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.979+0000 7f92ed8dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:56.968 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.979+0000 7f92e5ffb640 1 -- 172.21.15.104:0/2476327651 <== mon.1 v1:172.21.15.119:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92dc0017f0 con 0x7f92e8071eb0 2024-03-29T11:25:56.968 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.979+0000 7f92e5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:56.968 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.979+0000 7f92ed8dd640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:56.968 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.979+0000 7f92ed8dd640 4 set_mon_vals no callback set 2024-03-29T11:25:56.968 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.980+0000 7f92ed8dd640 10 monclient: shutdown 2024-03-29T11:25:56.968 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.980+0000 7f92ed8dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:56.968 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.980+0000 7f92ed8dd640 1 -- 172.21.15.104:0/2476327651 >> v1:172.21.15.119:6789/0 conn(0x7f92e8071eb0 legacy=0x7f92e8146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:56.969 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.980+0000 7f92ed8dd640 1 -- 172.21.15.104:0/2476327651 shutdown_connections 2024-03-29T11:25:56.969 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.980+0000 7f92ed8dd640 1 -- 172.21.15.104:0/2476327651 >> 172.21.15.104:0/2476327651 conn(0x7f92e8066a00 msgr2=0x7f92e8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:56.969 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.980+0000 7f92ed8dd640 1 -- 172.21.15.104:0/2476327651 shutdown_connections 2024-03-29T11:25:56.969 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.980+0000 7f92ed8dd640 1 -- 172.21.15.104:0/2476327651 wait complete. 2024-03-29T11:25:56.974 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.981+0000 7f92ed8dd640 10 monclient: build_initial_monmap 2024-03-29T11:25:56.975 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.981+0000 7f92ed8dd640 10 monclient: monmap: 2024-03-29T11:25:56.975 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:25:56.975 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:56.975 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:25:56.975 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:25:56.975 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:56.975 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:25:56.975 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:25:56.975 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:25:56.975 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:25:56.975 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:25:56.975 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.982+0000 7f92ed8dd640 1 librados: starting msgr at 2024-03-29T11:25:56.975 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.982+0000 7f92ed8dd640 1 librados: starting objecter 2024-03-29T11:25:56.975 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.982+0000 7f92ed8dd640 1 Processor -- start 2024-03-29T11:25:56.976 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.982+0000 7f92ed8dd640 1 -- start start 2024-03-29T11:25:56.976 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.982+0000 7f92ed8dd640 1 librados: setting wanted keys 2024-03-29T11:25:56.976 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.982+0000 7f92ed8dd640 1 librados: calling monclient init 2024-03-29T11:25:56.976 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.982+0000 7f92ed8dd640 10 monclient: init 2024-03-29T11:25:56.976 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.983+0000 7f92ed8dd640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:56.976 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.983+0000 7f92ed8dd640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:25:56.976 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.983+0000 7f92ed8dd640 10 monclient(hunting): picked mon.noname-b con 0x7f92e80b1f60 addr v1:172.21.15.119:6789/0 2024-03-29T11:25:56.976 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.983+0000 7f92ed8dd640 10 monclient(hunting): picked mon.noname-c con 0x7f92e8143d10 addr v1:172.21.15.104:6790/0 2024-03-29T11:25:56.976 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.983+0000 7f92ed8dd640 10 monclient(hunting): picked mon.noname-a con 0x7f92e8149290 addr v1:172.21.15.104:6789/0 2024-03-29T11:25:56.976 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.983+0000 7f92e77fe640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f92e8149290 0x7f92e81e5ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:39290/0 (socket says 172.21.15.104:39290) 2024-03-29T11:25:56.976 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.983+0000 7f92e77fe640 1 -- 172.21.15.104:0/655170504 learned_addr learned my addr 172.21.15.104:0/655170504 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:25:56.976 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.983+0000 7f92ed8dd640 1 -- 172.21.15.104:0/655170504 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92e814a640 con 0x7f92e8149290 2024-03-29T11:25:56.976 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.984+0000 7f92ed8dd640 1 -- 172.21.15.104:0/655170504 --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92e8149700 con 0x7f92e80b1f60 2024-03-29T11:25:56.976 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.984+0000 7f92ed8dd640 1 -- 172.21.15.104:0/655170504 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92e814a870 con 0x7f92e8143d10 2024-03-29T11:25:56.977 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.984+0000 7f92ed8dd640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:56.977 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.984+0000 7f92ed8dd640 10 monclient(hunting): authenticate will time out at 1338.235096s 2024-03-29T11:25:56.977 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.984+0000 7f92cffff640 10 client.?.objecter ms_handle_connect 0x7f92e8149290 2024-03-29T11:25:56.977 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.984+0000 7f92cffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:56.977 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.984+0000 7f92cffff640 10 client.?.objecter ms_handle_connect 0x7f92e80b1f60 2024-03-29T11:25:56.977 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.984+0000 7f92cffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:56.977 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.984+0000 7f92cffff640 10 client.?.objecter ms_handle_connect 0x7f92e8143d10 2024-03-29T11:25:56.977 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.984+0000 7f92cffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:56.977 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.984+0000 7f92cffff640 1 -- 172.21.15.104:0/655170504 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2163902356 0 0) 0x7f92e814a640 con 0x7f92e8149290 2024-03-29T11:25:56.977 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.984+0000 7f92cffff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:56.977 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.984+0000 7f92cffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:56.977 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.984+0000 7f92cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:56.977 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.984+0000 7f92cffff640 10 monclient(hunting): my global_id is 4476 2024-03-29T11:25:56.977 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.984+0000 7f92cffff640 1 -- 172.21.15.104:0/655170504 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92d4004eb0 con 0x7f92e8149290 2024-03-29T11:25:56.978 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.984+0000 7f92cffff640 1 -- 172.21.15.104:0/655170504 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3869417101 0 0) 0x7f92e8149700 con 0x7f92e80b1f60 2024-03-29T11:25:56.978 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.984+0000 7f92cffff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:56.978 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.984+0000 7f92cffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:56.978 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.984+0000 7f92cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:56.978 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.984+0000 7f92cffff640 10 monclient(hunting): my global_id is 4453 2024-03-29T11:25:56.978 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.984+0000 7f92cffff640 1 -- 172.21.15.104:0/655170504 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92e814a640 con 0x7f92e80b1f60 2024-03-29T11:25:56.978 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.985+0000 7f92cffff640 1 -- 172.21.15.104:0/655170504 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1022727014 0 0) 0x7f92e814a870 con 0x7f92e8143d10 2024-03-29T11:25:56.978 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.985+0000 7f92cffff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:56.978 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.985+0000 7f92cffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:56.978 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.985+0000 7f92cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:56.978 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.985+0000 7f92cffff640 10 monclient(hunting): my global_id is 4481 2024-03-29T11:25:56.978 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.985+0000 7f92cffff640 1 -- 172.21.15.104:0/655170504 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92d4005360 con 0x7f92e8143d10 2024-03-29T11:25:56.979 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.985+0000 7f92cffff640 1 -- 172.21.15.104:0/655170504 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 7334828 0 0) 0x7f92e814a640 con 0x7f92e80b1f60 2024-03-29T11:25:56.979 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.985+0000 7f92cffff640 1 -- 172.21.15.104:0/655170504 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92d40057b0 con 0x7f92e80b1f60 2024-03-29T11:25:56.979 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.985+0000 7f92cffff640 1 -- 172.21.15.104:0/655170504 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2443815754 0 0) 0x7f92d4004eb0 con 0x7f92e8149290 2024-03-29T11:25:56.979 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.985+0000 7f92cffff640 1 -- 172.21.15.104:0/655170504 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92e814a640 con 0x7f92e8149290 2024-03-29T11:25:56.979 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.986+0000 7f92cffff640 1 -- 172.21.15.104:0/655170504 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f92e0001ba0 con 0x7f92e80b1f60 2024-03-29T11:25:56.979 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.986+0000 7f92cffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:56.979 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.986+0000 7f92cffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:56.979 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.986+0000 7f92cffff640 10 monclient(hunting): dump: 2024-03-29T11:25:56.980 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:25:56.980 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:25:56.980 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:56.980 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:56.980 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:56.980 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:25:56.980 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:25:56.980 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:25:56.980 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:25:56.980 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:25:56.980 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.986+0000 7f92cffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:56.980 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.986+0000 7f92cffff640 1 -- 172.21.15.104:0/655170504 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f92d8001890 con 0x7f92e8149290 2024-03-29T11:25:56.980 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.986+0000 7f92cffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:56.980 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.986+0000 7f92cffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:56.981 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.986+0000 7f92cffff640 10 monclient(hunting): dump: 2024-03-29T11:25:56.981 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:25:56.981 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:25:56.981 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:56.981 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:56.981 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:56.981 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:25:56.981 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:25:56.981 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:25:56.981 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:25:56.981 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:25:56.981 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.986+0000 7f92cffff640 1 -- 172.21.15.104:0/655170504 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1237689794 0 0) 0x7f92d4005360 con 0x7f92e8143d10 2024-03-29T11:25:56.981 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.986+0000 7f92cffff640 1 -- 172.21.15.104:0/655170504 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92d4004eb0 con 0x7f92e8143d10 2024-03-29T11:25:56.981 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.986+0000 7f92cffff640 1 -- 172.21.15.104:0/655170504 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f92dc004980 con 0x7f92e8143d10 2024-03-29T11:25:56.981 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.986+0000 7f92cffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:56.982 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.986+0000 7f92cffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:56.982 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.986+0000 7f92cffff640 10 monclient(hunting): dump: 2024-03-29T11:25:56.982 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:25:56.982 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:25:56.982 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:56.982 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:56.982 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:56.982 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:25:56.982 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:25:56.982 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:25:56.982 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:25:56.982 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:25:56.982 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.987+0000 7f92cffff640 1 -- 172.21.15.104:0/655170504 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1935987196 0 0) 0x7f92d40057b0 con 0x7f92e80b1f60 2024-03-29T11:25:56.983 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.987+0000 7f92cffff640 1 -- 172.21.15.104:0/655170504 >> v1:172.21.15.104:6790/0 conn(0x7f92e8143d10 legacy=0x7f92e81548b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:56.983 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.987+0000 7f92cffff640 1 -- 172.21.15.104:0/655170504 >> v1:172.21.15.104:6789/0 conn(0x7f92e8149290 legacy=0x7f92e81e5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:56.983 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.987+0000 7f92cffff640 10 monclient: _finish_hunting 0 2024-03-29T11:25:56.983 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.987+0000 7f92cffff640 1 monclient: found mon.b 2024-03-29T11:25:56.983 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.987+0000 7f92cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:56.983 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.987+0000 7f92cffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:25:56.983 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.987+0000 7f92cffff640 1 -- 172.21.15.104:0/655170504 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92e814ab50 con 0x7f92e80b1f60 2024-03-29T11:25:56.983 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.987+0000 7f92cffff640 10 monclient: _finish_auth 0 2024-03-29T11:25:56.983 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.987+0000 7f92cffff640 10 monclient: _check_auth_tickets 2024-03-29T11:25:56.983 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.987+0000 7f92cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:56.983 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.987+0000 7f92ed8dd640 5 monclient: authenticate success, global_id 4453 2024-03-29T11:25:56.983 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.987+0000 7f92ed8dd640 10 monclient: _renew_subs 2024-03-29T11:25:56.983 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.987+0000 7f92ed8dd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:25:56.983 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.988+0000 7f92ed8dd640 1 -- 172.21.15.104:0/655170504 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92e814ae30 con 0x7f92e80b1f60 2024-03-29T11:25:56.983 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.988+0000 7f92ed8dd640 10 client.4453.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:56.984 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.988+0000 7f92ed8dd640 10 monclient: _renew_subs 2024-03-29T11:25:56.984 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.988+0000 7f92ed8dd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:25:56.984 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.988+0000 7f92ed8dd640 1 -- 172.21.15.104:0/655170504 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f92e81487d0 con 0x7f92e80b1f60 2024-03-29T11:25:56.984 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.988+0000 7f92ed8dd640 1 librados: init done 2024-03-29T11:25:56.984 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.989+0000 7f92cffff640 1 -- 172.21.15.104:0/655170504 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92e0003970 con 0x7f92e80b1f60 2024-03-29T11:25:56.984 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.989+0000 7f92cffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:56.984 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.989+0000 7f92e57fa640 4 set_mon_vals no callback set 2024-03-29T11:25:56.984 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.989+0000 7f92ed8dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:56.984 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.989+0000 7f92cffff640 1 -- 172.21.15.104:0/655170504 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f92e0001d00 con 0x7f92e80b1f60 2024-03-29T11:25:56.986 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.990+0000 7f92ed8dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:56.986 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.990+0000 7f92ed8dd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:25:56.986 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.990+0000 7f92ed8dd640 1 -- 172.21.15.104:0/655170504 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92c8005080 con 0x7f92e80b1f60 2024-03-29T11:25:56.986 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.990+0000 7f92cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:56.986 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.990+0000 7f92cffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:56.986 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.990+0000 7f92cffff640 10 monclient: dump: 2024-03-29T11:25:56.986 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:25:56.986 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:25:56.986 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:56.986 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:25:56.986 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:56.986 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:25:56.987 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:25:56.987 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:25:56.987 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:25:56.987 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:25:56.987 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.991+0000 7f92cffff640 1 -- 172.21.15.104:0/655170504 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f92e0003e00 con 0x7f92e80b1f60 2024-03-29T11:25:56.987 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.992+0000 7f92e67fc640 10 monclient: get_auth_request con 0x7f92d406e7e0 auth_method 0 2024-03-29T11:25:56.987 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.992+0000 7f92cffff640 1 -- 172.21.15.104:0/655170504 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f92e0083900 con 0x7f92e80b1f60 2024-03-29T11:25:56.987 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.992+0000 7f92cffff640 10 client.4453.objecter ms_dispatch 0x7f92e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:56.987 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.992+0000 7f92cffff640 3 client.4453.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:56.987 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.992+0000 7f92cffff640 3 client.4453.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:56.987 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.992+0000 7f92cffff640 20 client.4453.objecter dump_active .. 0 homeless 2024-03-29T11:25:56.987 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:56.992+0000 7f92cffff640 10 client.4453.objecter ms_handle_connect 0x7f92d406e7e0 2024-03-29T11:25:56.987 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:57.000+0000 7f92cffff640 1 -- 172.21.15.104:0/655170504 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f92e004f320 con 0x7f92e80b1f60 2024-03-29T11:25:56.987 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:57.000+0000 7f92cffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:56.988 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:57.000+0000 7f92cffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:57.376 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:57.390+0000 7f92ed8dd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:57.376 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:57.391+0000 7f92ed8dd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:57.376 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:57.391+0000 7f92ed8dd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:25:57.376 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:57.391+0000 7f92ed8dd640 1 -- 172.21.15.104:0/655170504 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92c8005330 con 0x7f92e80b1f60 2024-03-29T11:25:57.376 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:57.391+0000 7f92cffff640 1 -- 172.21.15.104:0/655170504 <== mon.1 v1:172.21.15.119:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f92e004f320 con 0x7f92e80b1f60 2024-03-29T11:25:57.377 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:57.391+0000 7f92cffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:57.377 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:57.391+0000 7f92cffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:57.377 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:25:57.377 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:57.381 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:57.395+0000 7f92ed8dd640 10 librados: watch_flush enter 2024-03-29T11:25:57.381 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:57.395+0000 7f92ed8dd640 10 librados: watch_flush exit 2024-03-29T11:25:57.381 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:57.395+0000 7f92ed8dd640 20 client.4453.objecter shutdown clearing up homeless session... 2024-03-29T11:25:57.381 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:57.395+0000 7f92ed8dd640 10 client.4453.objecter successfully canceled tick 2024-03-29T11:25:57.381 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:57.395+0000 7f92ed8dd640 1 -- 172.21.15.104:0/655170504 >> v1:172.21.15.104:6816/3341126660 conn(0x7f92d406e7e0 legacy=0x7f92d4070ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:57.382 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:57.396+0000 7f92ed8dd640 10 monclient: shutdown 2024-03-29T11:25:57.382 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:57.396+0000 7f92ed8dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:57.382 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:57.396+0000 7f92ed8dd640 1 -- 172.21.15.104:0/655170504 >> v1:172.21.15.119:6789/0 conn(0x7f92e80b1f60 legacy=0x7f92e81541a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:57.382 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:57.396+0000 7f92ed8dd640 1 -- 172.21.15.104:0/655170504 shutdown_connections 2024-03-29T11:25:57.382 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:57.396+0000 7f92ed8dd640 1 -- 172.21.15.104:0/655170504 >> 172.21.15.104:0/655170504 conn(0x7f92e8066a00 msgr2=0x7f92e81468a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:57.382 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:57.396+0000 7f92ed8dd640 1 -- 172.21.15.104:0/655170504 shutdown_connections 2024-03-29T11:25:57.382 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:57.396+0000 7f92ed8dd640 1 -- 172.21.15.104:0/655170504 wait complete. 2024-03-29T11:25:57.382 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:25:57.397+0000 7f92ed8dd640 1 librados: shutdown 2024-03-29T11:25:57.397 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:04.205 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:04.350 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.364+0000 7f59be9db640 10 monclient: get_monmap_and_config 2024-03-29T11:26:04.350 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.364+0000 7f59be9db640 10 monclient: build_initial_monmap 2024-03-29T11:26:04.350 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.364+0000 7f59be9db640 10 monclient: monmap: 2024-03-29T11:26:04.350 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:26:04.350 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:26:04.350 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:26:04.365564+0000 2024-03-29T11:26:04.350 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:26:04.365564+0000 2024-03-29T11:26:04.350 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:04.350 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:26:04.350 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:26:04.351 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:26:04.351 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:26:04.351 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:26:04.352 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.365+0000 7f59be9db640 1 Processor -- start 2024-03-29T11:26:04.352 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.365+0000 7f59be9db640 1 -- start start 2024-03-29T11:26:04.352 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.365+0000 7f59be9db640 10 monclient: init 2024-03-29T11:26:04.352 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.366+0000 7f59be9db640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:04.352 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.366+0000 7f59be9db640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:26:04.352 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.366+0000 7f59be9db640 10 monclient(hunting): picked mon.noname-b con 0x7f59b81422b0 addr v1:172.21.15.104:6790/0 2024-03-29T11:26:04.352 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.366+0000 7f59be9db640 10 monclient(hunting): picked mon.noname-c con 0x7f59b8142690 addr v1:172.21.15.119:6789/0 2024-03-29T11:26:04.352 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.366+0000 7f59be9db640 10 monclient(hunting): picked mon.noname-a con 0x7f59b80b17d0 addr v1:172.21.15.104:6789/0 2024-03-29T11:26:04.352 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.366+0000 7f59be9db640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59b8142c80 con 0x7f59b80b17d0 2024-03-29T11:26:04.352 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.366+0000 7f59be9db640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59b80aae40 con 0x7f59b8142690 2024-03-29T11:26:04.352 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.366+0000 7f59be9db640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59b813fbd0 con 0x7f59b81422b0 2024-03-29T11:26:04.352 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.366+0000 7f59be9db640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:04.353 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.366+0000 7f59be9db640 10 monclient(hunting): authenticate will time out at 1345.617719s 2024-03-29T11:26:04.353 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.367+0000 7f59bcf78640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f59b80b17d0 0x7f59b80b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:43090/0 (socket says 172.21.15.104:43090) 2024-03-29T11:26:04.353 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.367+0000 7f59bcf78640 1 -- 172.21.15.104:0/2647421632 learned_addr learned my addr 172.21.15.104:0/2647421632 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:26:04.354 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.367+0000 7f59b6ffd640 1 -- 172.21.15.104:0/2647421632 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1636286169 0 0) 0x7f59b8142c80 con 0x7f59b80b17d0 2024-03-29T11:26:04.354 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.367+0000 7f59b6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:04.354 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.368+0000 7f59b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:04.354 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.368+0000 7f59b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:04.354 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.368+0000 7f59b6ffd640 10 monclient(hunting): my global_id is 4479 2024-03-29T11:26:04.354 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.368+0000 7f59b6ffd640 1 -- 172.21.15.104:0/2647421632 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59a0003520 con 0x7f59b80b17d0 2024-03-29T11:26:04.354 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.368+0000 7f59b6ffd640 1 -- 172.21.15.104:0/2647421632 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3803892047 0 0) 0x7f59b80aae40 con 0x7f59b8142690 2024-03-29T11:26:04.354 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.368+0000 7f59b6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:04.354 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.368+0000 7f59b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:04.354 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.368+0000 7f59b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:04.355 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.368+0000 7f59b6ffd640 10 monclient(hunting): my global_id is 4456 2024-03-29T11:26:04.355 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.368+0000 7f59b6ffd640 1 -- 172.21.15.104:0/2647421632 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59b8142c80 con 0x7f59b8142690 2024-03-29T11:26:04.355 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.368+0000 7f59b6ffd640 1 -- 172.21.15.104:0/2647421632 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3572772659 0 0) 0x7f59b813fbd0 con 0x7f59b81422b0 2024-03-29T11:26:04.355 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.368+0000 7f59b6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:04.355 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.368+0000 7f59b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:04.355 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.368+0000 7f59b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:04.355 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.368+0000 7f59b6ffd640 10 monclient(hunting): my global_id is 4484 2024-03-29T11:26:04.355 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.368+0000 7f59b6ffd640 1 -- 172.21.15.104:0/2647421632 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59b80aae40 con 0x7f59b81422b0 2024-03-29T11:26:04.355 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.368+0000 7f59b6ffd640 1 -- 172.21.15.104:0/2647421632 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 456332293 0 0) 0x7f59b8142c80 con 0x7f59b8142690 2024-03-29T11:26:04.355 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.368+0000 7f59b6ffd640 1 -- 172.21.15.104:0/2647421632 >> v1:172.21.15.104:6790/0 conn(0x7f59b81422b0 legacy=0x7f59b813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:04.355 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.368+0000 7f59b6ffd640 1 -- 172.21.15.104:0/2647421632 >> v1:172.21.15.104:6789/0 conn(0x7f59b80b17d0 legacy=0x7f59b80b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:04.355 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.368+0000 7f59b6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:26:04.355 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.368+0000 7f59b6ffd640 1 monclient: found mon.noname-c 2024-03-29T11:26:04.355 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.368+0000 7f59b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:04.356 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.368+0000 7f59b6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.119:6789/0 2024-03-29T11:26:04.356 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.368+0000 7f59b6ffd640 1 -- 172.21.15.104:0/2647421632 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b8151940 con 0x7f59b8142690 2024-03-29T11:26:04.356 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.368+0000 7f59b6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:26:04.356 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.368+0000 7f59b6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:26:04.356 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.368+0000 7f59b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:04.356 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.368+0000 7f59be9db640 5 monclient: authenticate success, global_id 4456 2024-03-29T11:26:04.356 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.368+0000 7f59be9db640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:04.356 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.368+0000 7f59b6ffd640 1 -- 172.21.15.104:0/2647421632 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f59a8001670 con 0x7f59b8142690 2024-03-29T11:26:04.356 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.368+0000 7f59b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:04.356 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.368+0000 7f59b6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:04.356 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.368+0000 7f59b6ffd640 10 monclient: dump: 2024-03-29T11:26:04.356 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:26:04.356 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:26:04.356 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:04.356 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:04.357 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:04.357 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:26:04.357 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:26:04.357 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:26:04.357 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:26:04.357 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:26:04.357 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.368+0000 7f59be9db640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:04.357 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.368+0000 7f59b6ffd640 1 -- 172.21.15.104:0/2647421632 <== mon.1 v1:172.21.15.119:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59a80017d0 con 0x7f59b8142690 2024-03-29T11:26:04.357 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.368+0000 7f59b6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:04.357 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.368+0000 7f59be9db640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:04.357 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.368+0000 7f59be9db640 4 set_mon_vals no callback set 2024-03-29T11:26:04.358 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.369+0000 7f59b6ffd640 1 -- 172.21.15.104:0/2647421632 <== mon.1 v1:172.21.15.119:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f59a8001b60 con 0x7f59b8142690 2024-03-29T11:26:04.358 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.369+0000 7f59b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:04.358 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.369+0000 7f59b6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:04.358 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.369+0000 7f59b6ffd640 10 monclient: dump: 2024-03-29T11:26:04.358 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:26:04.358 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:26:04.358 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:04.358 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:04.358 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:04.358 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:26:04.358 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:26:04.358 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:26:04.358 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:26:04.358 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:26:04.364 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.369+0000 7f59be9db640 10 monclient: shutdown 2024-03-29T11:26:04.365 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.369+0000 7f59be9db640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:04.365 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.369+0000 7f59be9db640 1 -- 172.21.15.104:0/2647421632 >> v1:172.21.15.119:6789/0 conn(0x7f59b8142690 legacy=0x7f59b80b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:04.365 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.369+0000 7f59be9db640 1 -- 172.21.15.104:0/2647421632 shutdown_connections 2024-03-29T11:26:04.365 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.369+0000 7f59be9db640 1 -- 172.21.15.104:0/2647421632 >> 172.21.15.104:0/2647421632 conn(0x7f59b8066a00 msgr2=0x7f59b8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:04.365 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.370+0000 7f59be9db640 1 -- 172.21.15.104:0/2647421632 shutdown_connections 2024-03-29T11:26:04.365 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.370+0000 7f59be9db640 1 -- 172.21.15.104:0/2647421632 wait complete. 2024-03-29T11:26:04.365 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.371+0000 7f59be9db640 10 monclient: build_initial_monmap 2024-03-29T11:26:04.365 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.371+0000 7f59be9db640 10 monclient: monmap: 2024-03-29T11:26:04.365 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:26:04.365 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:04.365 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:26:04.365 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:26:04.365 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:04.366 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:26:04.366 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:26:04.366 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:26:04.366 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:26:04.366 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:26:04.366 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.371+0000 7f59be9db640 1 librados: starting msgr at 2024-03-29T11:26:04.366 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.371+0000 7f59be9db640 1 librados: starting objecter 2024-03-29T11:26:04.366 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.371+0000 7f59be9db640 1 Processor -- start 2024-03-29T11:26:04.366 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.371+0000 7f59be9db640 1 -- start start 2024-03-29T11:26:04.366 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.371+0000 7f59be9db640 1 librados: setting wanted keys 2024-03-29T11:26:04.366 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.371+0000 7f59be9db640 1 librados: calling monclient init 2024-03-29T11:26:04.366 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.371+0000 7f59be9db640 10 monclient: init 2024-03-29T11:26:04.366 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.372+0000 7f59be9db640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:04.366 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.372+0000 7f59be9db640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:26:04.367 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.372+0000 7f59be9db640 10 monclient(hunting): picked mon.noname-a con 0x7f59b80b17d0 addr v1:172.21.15.104:6789/0 2024-03-29T11:26:04.367 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.372+0000 7f59be9db640 10 monclient(hunting): picked mon.noname-c con 0x7f59b81422b0 addr v1:172.21.15.104:6790/0 2024-03-29T11:26:04.367 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.372+0000 7f59be9db640 10 monclient(hunting): picked mon.noname-b con 0x7f59b8142690 addr v1:172.21.15.119:6789/0 2024-03-29T11:26:04.367 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.372+0000 7f59be9db640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59b806ed40 con 0x7f59b80b17d0 2024-03-29T11:26:04.367 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.372+0000 7f59be9db640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59b8132710 con 0x7f59b8142690 2024-03-29T11:26:04.367 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.372+0000 7f59be9db640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59b81322c0 con 0x7f59b81422b0 2024-03-29T11:26:04.367 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.372+0000 7f59be9db640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:04.367 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.372+0000 7f59be9db640 10 monclient(hunting): authenticate will time out at 1345.623795s 2024-03-29T11:26:04.367 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.372+0000 7f59bcf78640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f59b8142690 0x7f59b81df360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:33274/0 (socket says 172.21.15.104:33274) 2024-03-29T11:26:04.367 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.372+0000 7f59bcf78640 1 -- 172.21.15.104:0/1551020169 learned_addr learned my addr 172.21.15.104:0/1551020169 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:26:04.367 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.373+0000 7f59b4ff9640 10 client.?.objecter ms_handle_connect 0x7f59b8142690 2024-03-29T11:26:04.367 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.373+0000 7f59b4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:04.367 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.373+0000 7f59b4ff9640 10 client.?.objecter ms_handle_connect 0x7f59b80b17d0 2024-03-29T11:26:04.368 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.373+0000 7f59b4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:04.368 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.373+0000 7f59b4ff9640 10 client.?.objecter ms_handle_connect 0x7f59b81422b0 2024-03-29T11:26:04.368 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.373+0000 7f59b4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:04.368 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.373+0000 7f59b4ff9640 1 -- 172.21.15.104:0/1551020169 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4123115716 0 0) 0x7f59b806ed40 con 0x7f59b80b17d0 2024-03-29T11:26:04.368 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.373+0000 7f59b4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:26:04.368 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.373+0000 7f59b4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:04.368 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.373+0000 7f59b4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:04.368 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.373+0000 7f59b4ff9640 10 monclient(hunting): my global_id is 4482 2024-03-29T11:26:04.368 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.373+0000 7f59b4ff9640 1 -- 172.21.15.104:0/1551020169 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59a4004eb0 con 0x7f59b80b17d0 2024-03-29T11:26:04.368 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.373+0000 7f59b4ff9640 1 -- 172.21.15.104:0/1551020169 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2704416408 0 0) 0x7f59b81322c0 con 0x7f59b81422b0 2024-03-29T11:26:04.368 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.373+0000 7f59b4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:26:04.368 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.373+0000 7f59b4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:04.368 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.373+0000 7f59b4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:04.368 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.373+0000 7f59b4ff9640 10 monclient(hunting): my global_id is 4487 2024-03-29T11:26:04.368 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.373+0000 7f59b4ff9640 1 -- 172.21.15.104:0/1551020169 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59a40051d0 con 0x7f59b81422b0 2024-03-29T11:26:04.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.374+0000 7f59b4ff9640 1 -- 172.21.15.104:0/1551020169 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 920519474 0 0) 0x7f59a40051d0 con 0x7f59b81422b0 2024-03-29T11:26:04.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.374+0000 7f59b4ff9640 1 -- 172.21.15.104:0/1551020169 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59b81322c0 con 0x7f59b81422b0 2024-03-29T11:26:04.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.374+0000 7f59b4ff9640 1 -- 172.21.15.104:0/1551020169 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f59a8001ae0 con 0x7f59b81422b0 2024-03-29T11:26:04.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.374+0000 7f59b4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:04.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.374+0000 7f59b4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:04.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.374+0000 7f59b4ff9640 10 monclient(hunting): dump: 2024-03-29T11:26:04.369 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:26:04.369 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:26:04.369 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:04.369 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:04.369 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:04.369 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:26:04.369 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:26:04.369 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:26:04.369 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:26:04.370 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:26:04.370 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.374+0000 7f59b4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:04.370 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.374+0000 7f59b4ff9640 1 -- 172.21.15.104:0/1551020169 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3151024248 0 0) 0x7f59a4004eb0 con 0x7f59b80b17d0 2024-03-29T11:26:04.370 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.374+0000 7f59b4ff9640 1 -- 172.21.15.104:0/1551020169 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59a40051d0 con 0x7f59b80b17d0 2024-03-29T11:26:04.370 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.374+0000 7f59b4ff9640 1 -- 172.21.15.104:0/1551020169 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f59ac001930 con 0x7f59b80b17d0 2024-03-29T11:26:04.370 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.374+0000 7f59b4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:04.370 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.374+0000 7f59b4ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:04.370 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.374+0000 7f59b4ff9640 10 monclient(hunting): dump: 2024-03-29T11:26:04.370 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:26:04.370 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:26:04.370 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:04.370 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:04.371 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:04.371 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:26:04.371 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:26:04.371 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:26:04.371 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:26:04.371 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:26:04.371 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.374+0000 7f59b4ff9640 1 -- 172.21.15.104:0/1551020169 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1795045397 0 0) 0x7f59b8132710 con 0x7f59b8142690 2024-03-29T11:26:04.371 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.374+0000 7f59b4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:26:04.371 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.374+0000 7f59b4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:04.371 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.374+0000 7f59b4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:04.371 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.374+0000 7f59b4ff9640 10 monclient(hunting): my global_id is 4459 2024-03-29T11:26:04.371 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.374+0000 7f59b4ff9640 1 -- 172.21.15.104:0/1551020169 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59a4004eb0 con 0x7f59b8142690 2024-03-29T11:26:04.371 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.374+0000 7f59b4ff9640 1 -- 172.21.15.104:0/1551020169 <== mon.2 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3933326852 0 0) 0x7f59b81322c0 con 0x7f59b81422b0 2024-03-29T11:26:04.371 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.375+0000 7f59b4ff9640 1 -- 172.21.15.104:0/1551020169 >> v1:172.21.15.119:6789/0 conn(0x7f59b8142690 legacy=0x7f59b81df360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:04.371 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.375+0000 7f59b4ff9640 1 -- 172.21.15.104:0/1551020169 >> v1:172.21.15.104:6789/0 conn(0x7f59b80b17d0 legacy=0x7f59b812ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:04.372 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.375+0000 7f59b4ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:26:04.372 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.375+0000 7f59b4ff9640 1 monclient: found mon.c 2024-03-29T11:26:04.372 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.375+0000 7f59b4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:04.372 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.375+0000 7f59b4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:26:04.372 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.375+0000 7f59b4ff9640 1 -- 172.21.15.104:0/1551020169 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b8132450 con 0x7f59b81422b0 2024-03-29T11:26:04.372 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.375+0000 7f59b4ff9640 10 monclient: _finish_auth 0 2024-03-29T11:26:04.372 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.375+0000 7f59b4ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:26:04.372 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.375+0000 7f59b4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:04.372 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.375+0000 7f59be9db640 5 monclient: authenticate success, global_id 4487 2024-03-29T11:26:04.372 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.375+0000 7f59be9db640 10 monclient: _renew_subs 2024-03-29T11:26:04.372 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.375+0000 7f59be9db640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:26:04.372 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.375+0000 7f59be9db640 1 -- 172.21.15.104:0/1551020169 --> v1:172.21.15.104:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59b8131890 con 0x7f59b81422b0 2024-03-29T11:26:04.372 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.375+0000 7f59be9db640 10 client.4487.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:04.372 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.375+0000 7f59be9db640 10 monclient: _renew_subs 2024-03-29T11:26:04.372 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.375+0000 7f59be9db640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:26:04.373 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.375+0000 7f59be9db640 1 -- 172.21.15.104:0/1551020169 --> v1:172.21.15.104:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f59b813bb10 con 0x7f59b81422b0 2024-03-29T11:26:04.373 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.375+0000 7f59be9db640 1 librados: init done 2024-03-29T11:26:04.373 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.375+0000 7f59b4ff9640 1 -- 172.21.15.104:0/1551020169 <== mon.2 v1:172.21.15.104:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59a8001ca0 con 0x7f59b81422b0 2024-03-29T11:26:04.373 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.375+0000 7f59b4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:04.373 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.375+0000 7f59b4ff9640 1 -- 172.21.15.104:0/1551020169 <== mon.2 v1:172.21.15.104:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f59a8003a40 con 0x7f59b81422b0 2024-03-29T11:26:04.373 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.375+0000 7f59b4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:04.373 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.375+0000 7f59b4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:04.373 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.375+0000 7f59b4ff9640 10 monclient: dump: 2024-03-29T11:26:04.373 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:26:04.373 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:26:04.373 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:04.373 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:04.373 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:04.374 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:26:04.374 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:26:04.374 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:26:04.374 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:26:04.374 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:26:04.374 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.375+0000 7f59b5ffb640 4 set_mon_vals no callback set 2024-03-29T11:26:04.374 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.376+0000 7f599e7fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:04.374 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.376+0000 7f599e7fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:04.374 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.376+0000 7f599e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:26:04.374 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.376+0000 7f599e7fc640 1 -- 172.21.15.104:0/1551020169 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5990005080 con 0x7f59b81422b0 2024-03-29T11:26:04.374 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.377+0000 7f59b4ff9640 1 -- 172.21.15.104:0/1551020169 <== mon.2 v1:172.21.15.104:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f59a8019a10 con 0x7f59b81422b0 2024-03-29T11:26:04.374 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.378+0000 7f59b4ff9640 1 -- 172.21.15.104:0/1551020169 <== mon.2 v1:172.21.15.104:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f59a8083830 con 0x7f59b81422b0 2024-03-29T11:26:04.374 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.378+0000 7f59b4ff9640 10 client.4487.objecter ms_dispatch 0x7f59b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:04.375 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.378+0000 7f59b4ff9640 3 client.4487.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:04.375 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.378+0000 7f59b4ff9640 3 client.4487.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:04.375 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.378+0000 7f59b7fff640 10 monclient: get_auth_request con 0x7f59a406e110 auth_method 0 2024-03-29T11:26:04.375 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.378+0000 7f59b4ff9640 20 client.4487.objecter dump_active .. 0 homeless 2024-03-29T11:26:04.375 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.379+0000 7f59b4ff9640 10 client.4487.objecter ms_handle_connect 0x7f59a406e110 2024-03-29T11:26:04.375 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.386+0000 7f59b4ff9640 1 -- 172.21.15.104:0/1551020169 <== mon.2 v1:172.21.15.104:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f59a804f170 con 0x7f59b81422b0 2024-03-29T11:26:04.375 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.386+0000 7f59b4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:04.376 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.386+0000 7f59b4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:04.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.760+0000 7f599e7fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:04.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.760+0000 7f599e7fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:04.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.760+0000 7f599e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:26:04.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.760+0000 7f599e7fc640 1 -- 172.21.15.104:0/1551020169 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5990005330 con 0x7f59b81422b0 2024-03-29T11:26:04.747 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.761+0000 7f59b4ff9640 1 -- 172.21.15.104:0/1551020169 <== mon.2 v1:172.21.15.104:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f59a8082ee0 con 0x7f59b81422b0 2024-03-29T11:26:04.747 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.761+0000 7f59b4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:04.747 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.761+0000 7f59b4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:04.748 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:26:04.748 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:04.751 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.765+0000 7f599e7fc640 10 librados: watch_flush enter 2024-03-29T11:26:04.751 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.765+0000 7f599e7fc640 10 librados: watch_flush exit 2024-03-29T11:26:04.751 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.765+0000 7f599e7fc640 20 client.4487.objecter shutdown clearing up homeless session... 2024-03-29T11:26:04.751 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.765+0000 7f599e7fc640 10 client.4487.objecter successfully canceled tick 2024-03-29T11:26:04.751 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.765+0000 7f599e7fc640 1 -- 172.21.15.104:0/1551020169 >> v1:172.21.15.104:6816/3341126660 conn(0x7f59a406e110 legacy=0x7f59a40705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:04.752 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.765+0000 7f599e7fc640 10 monclient: shutdown 2024-03-29T11:26:04.752 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.765+0000 7f599e7fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:04.752 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.765+0000 7f599e7fc640 1 -- 172.21.15.104:0/1551020169 >> v1:172.21.15.104:6790/0 conn(0x7f59b81422b0 legacy=0x7f59b81dec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:04.752 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.765+0000 7f599e7fc640 1 -- 172.21.15.104:0/1551020169 shutdown_connections 2024-03-29T11:26:04.752 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.765+0000 7f599e7fc640 1 -- 172.21.15.104:0/1551020169 >> 172.21.15.104:0/1551020169 conn(0x7f59b8066a00 msgr2=0x7f59b80b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:04.752 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.765+0000 7f599e7fc640 1 -- 172.21.15.104:0/1551020169 shutdown_connections 2024-03-29T11:26:04.753 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.766+0000 7f599e7fc640 1 -- 172.21.15.104:0/1551020169 wait complete. 2024-03-29T11:26:04.753 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:04.766+0000 7f599e7fc640 1 librados: shutdown 2024-03-29T11:26:04.766 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:11.572 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:11.718 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.731+0000 7f4bc85ba640 10 monclient: get_monmap_and_config 2024-03-29T11:26:11.718 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.731+0000 7f4bc85ba640 10 monclient: build_initial_monmap 2024-03-29T11:26:11.718 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.731+0000 7f4bc85ba640 10 monclient: monmap: 2024-03-29T11:26:11.718 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:26:11.718 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:26:11.718 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:26:11.732880+0000 2024-03-29T11:26:11.718 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:26:11.732880+0000 2024-03-29T11:26:11.718 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:11.718 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:26:11.719 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:26:11.719 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:26:11.719 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:26:11.719 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:26:11.720 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.734+0000 7f4bc85ba640 1 Processor -- start 2024-03-29T11:26:11.720 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.734+0000 7f4bc85ba640 1 -- start start 2024-03-29T11:26:11.720 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.734+0000 7f4bc85ba640 10 monclient: init 2024-03-29T11:26:11.721 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.735+0000 7f4bc85ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:11.721 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.735+0000 7f4bc85ba640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:26:11.721 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.735+0000 7f4bc85ba640 10 monclient(hunting): picked mon.noname-c con 0x7f4bc00a9930 addr v1:172.21.15.119:6789/0 2024-03-29T11:26:11.721 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.735+0000 7f4bc85ba640 10 monclient(hunting): picked mon.noname-b con 0x7f4bc0149dc0 addr v1:172.21.15.104:6790/0 2024-03-29T11:26:11.721 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.735+0000 7f4bc85ba640 10 monclient(hunting): picked mon.noname-a con 0x7f4bc014ff40 addr v1:172.21.15.104:6789/0 2024-03-29T11:26:11.721 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.735+0000 7f4bc85ba640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bc00aae60 con 0x7f4bc014ff40 2024-03-29T11:26:11.721 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.735+0000 7f4bc85ba640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bc00b4310 con 0x7f4bc00a9930 2024-03-29T11:26:11.721 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.735+0000 7f4bc85ba640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bc00afdd0 con 0x7f4bc0149dc0 2024-03-29T11:26:11.721 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.735+0000 7f4bc85ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:11.721 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.735+0000 7f4bc85ba640 10 monclient(hunting): authenticate will time out at 1352.986380s 2024-03-29T11:26:11.722 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.736+0000 7f4bc5b55640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f4bc0149dc0 0x7f4bc014a240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:41020/0 (socket says 172.21.15.104:41020) 2024-03-29T11:26:11.722 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.736+0000 7f4bc5b55640 1 -- 172.21.15.104:0/42934314 learned_addr learned my addr 172.21.15.104:0/42934314 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:26:11.724 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.738+0000 7f4bc5354640 1 -- 172.21.15.104:0/42934314 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3376633489 0 0) 0x7f4bc00afdd0 con 0x7f4bc0149dc0 2024-03-29T11:26:11.724 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.738+0000 7f4bc5354640 10 monclient(hunting): _negotiate 2024-03-29T11:26:11.724 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.738+0000 7f4bc5354640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:11.724 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.738+0000 7f4bc5354640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:11.724 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.738+0000 7f4bc5354640 10 monclient(hunting): my global_id is 4490 2024-03-29T11:26:11.724 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.738+0000 7f4bc5354640 1 -- 172.21.15.104:0/42934314 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bac003520 con 0x7f4bc0149dc0 2024-03-29T11:26:11.724 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.739+0000 7f4bc5354640 1 -- 172.21.15.104:0/42934314 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2064048226 0 0) 0x7f4bc00aae60 con 0x7f4bc014ff40 2024-03-29T11:26:11.724 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.739+0000 7f4bc5354640 10 monclient(hunting): _negotiate 2024-03-29T11:26:11.724 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.739+0000 7f4bc5354640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:11.724 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.739+0000 7f4bc5354640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:11.725 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.739+0000 7f4bc5354640 10 monclient(hunting): my global_id is 4485 2024-03-29T11:26:11.725 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.739+0000 7f4bc5354640 1 -- 172.21.15.104:0/42934314 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bc00afdd0 con 0x7f4bc014ff40 2024-03-29T11:26:11.726 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.739+0000 7f4bc5354640 1 -- 172.21.15.104:0/42934314 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3005096453 0 0) 0x7f4bc00afdd0 con 0x7f4bc014ff40 2024-03-29T11:26:11.726 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.739+0000 7f4bc5354640 1 -- 172.21.15.104:0/42934314 >> v1:172.21.15.104:6790/0 conn(0x7f4bc0149dc0 legacy=0x7f4bc014a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:11.726 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.739+0000 7f4bc5354640 1 -- 172.21.15.104:0/42934314 >> v1:172.21.15.119:6789/0 conn(0x7f4bc00a9930 legacy=0x7f4bc0143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:26:11.726 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.739+0000 7f4bc5354640 10 monclient: _finish_hunting 0 2024-03-29T11:26:11.726 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.739+0000 7f4bc5354640 1 monclient: found mon.noname-a 2024-03-29T11:26:11.726 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.739+0000 7f4bc5354640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:11.726 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.739+0000 7f4bc5354640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.104:6789/0 2024-03-29T11:26:11.727 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.739+0000 7f4bc5354640 1 -- 172.21.15.104:0/42934314 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc014a950 con 0x7f4bc014ff40 2024-03-29T11:26:11.727 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.739+0000 7f4bc5354640 10 monclient: _finish_auth 0 2024-03-29T11:26:11.727 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.739+0000 7f4bc5354640 10 monclient: _check_auth_tickets 2024-03-29T11:26:11.727 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.739+0000 7f4bc5354640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:11.727 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.739+0000 7f4bc85ba640 5 monclient: authenticate success, global_id 4485 2024-03-29T11:26:11.727 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.739+0000 7f4bc85ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:11.727 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.739+0000 7f4bc5354640 1 -- 172.21.15.104:0/42934314 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f4bb80019f0 con 0x7f4bc014ff40 2024-03-29T11:26:11.727 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.739+0000 7f4bc5354640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:11.727 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.739+0000 7f4bc5354640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:11.727 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.739+0000 7f4bc5354640 10 monclient: dump: 2024-03-29T11:26:11.727 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:26:11.727 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:26:11.727 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:11.727 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:11.728 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:11.728 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:26:11.728 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:26:11.728 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:26:11.728 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:26:11.728 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:26:11.728 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.739+0000 7f4bc85ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:11.728 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.739+0000 7f4bc5354640 1 -- 172.21.15.104:0/42934314 <== mon.0 v1:172.21.15.104:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4bb8001c40 con 0x7f4bc014ff40 2024-03-29T11:26:11.728 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.739+0000 7f4bc5354640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:11.728 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.739+0000 7f4bc85ba640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:11.728 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.739+0000 7f4bc85ba640 4 set_mon_vals no callback set 2024-03-29T11:26:11.728 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.740+0000 7f4bc85ba640 10 monclient: shutdown 2024-03-29T11:26:11.728 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.740+0000 7f4bc85ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:11.728 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.740+0000 7f4bc85ba640 1 -- 172.21.15.104:0/42934314 >> v1:172.21.15.104:6789/0 conn(0x7f4bc014ff40 legacy=0x7f4bc0152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:11.729 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.742+0000 7f4bc85ba640 1 -- 172.21.15.104:0/42934314 shutdown_connections 2024-03-29T11:26:11.729 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.742+0000 7f4bc85ba640 1 -- 172.21.15.104:0/42934314 >> 172.21.15.104:0/42934314 conn(0x7f4bc00b35e0 msgr2=0x7f4bc00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:11.729 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.742+0000 7f4bc85ba640 1 -- 172.21.15.104:0/42934314 shutdown_connections 2024-03-29T11:26:11.729 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.742+0000 7f4bc85ba640 1 -- 172.21.15.104:0/42934314 wait complete. 2024-03-29T11:26:11.730 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.744+0000 7f4bc85ba640 10 monclient: build_initial_monmap 2024-03-29T11:26:11.730 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.744+0000 7f4bc85ba640 10 monclient: monmap: 2024-03-29T11:26:11.730 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:26:11.730 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:11.730 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:26:11.730 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:26:11.730 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:11.730 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:26:11.731 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:26:11.731 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:26:11.731 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:26:11.731 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:26:11.731 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.744+0000 7f4bc85ba640 1 librados: starting msgr at 2024-03-29T11:26:11.731 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.744+0000 7f4bc85ba640 1 librados: starting objecter 2024-03-29T11:26:11.731 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.745+0000 7f4bc85ba640 1 Processor -- start 2024-03-29T11:26:11.732 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.745+0000 7f4bc85ba640 1 -- start start 2024-03-29T11:26:11.733 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.745+0000 7f4bc85ba640 1 librados: setting wanted keys 2024-03-29T11:26:11.733 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.745+0000 7f4bc85ba640 1 librados: calling monclient init 2024-03-29T11:26:11.733 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.745+0000 7f4bc85ba640 10 monclient: init 2024-03-29T11:26:11.733 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.745+0000 7f4bc85ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:11.733 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.745+0000 7f4bc85ba640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:26:11.733 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.745+0000 7f4bc85ba640 10 monclient(hunting): picked mon.noname-a con 0x7f4bc00a9930 addr v1:172.21.15.104:6789/0 2024-03-29T11:26:11.733 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.745+0000 7f4bc85ba640 10 monclient(hunting): picked mon.noname-c con 0x7f4bc0149dc0 addr v1:172.21.15.104:6790/0 2024-03-29T11:26:11.733 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.745+0000 7f4bc85ba640 10 monclient(hunting): picked mon.noname-b con 0x7f4bc014ff40 addr v1:172.21.15.119:6789/0 2024-03-29T11:26:11.733 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.745+0000 7f4bc85ba640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bc013f7f0 con 0x7f4bc00a9930 2024-03-29T11:26:11.733 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.745+0000 7f4bc85ba640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bc00ab760 con 0x7f4bc014ff40 2024-03-29T11:26:11.733 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.745+0000 7f4bc85ba640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bc00ab430 con 0x7f4bc0149dc0 2024-03-29T11:26:11.733 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.745+0000 7f4bc85ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:11.733 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.745+0000 7f4bc85ba640 10 monclient(hunting): authenticate will time out at 1352.996992s 2024-03-29T11:26:11.735 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.747+0000 7f4bc5b55640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f4bc0149dc0 0x7f4bc0071a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:50152/0 (socket says 172.21.15.104:50152) 2024-03-29T11:26:11.735 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.747+0000 7f4bc5b55640 1 -- 172.21.15.104:0/2878322893 learned_addr learned my addr 172.21.15.104:0/2878322893 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:26:11.735 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.748+0000 7f4bb6ffd640 10 client.?.objecter ms_handle_connect 0x7f4bc0149dc0 2024-03-29T11:26:11.735 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.748+0000 7f4bb6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:11.735 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.748+0000 7f4bb6ffd640 1 -- 172.21.15.104:0/2878322893 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3787497095 0 0) 0x7f4bc00ab430 con 0x7f4bc0149dc0 2024-03-29T11:26:11.735 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.748+0000 7f4bb6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:11.735 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.748+0000 7f4bb6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:11.735 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.748+0000 7f4bb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:11.735 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.748+0000 7f4bb6ffd640 10 monclient(hunting): my global_id is 4493 2024-03-29T11:26:11.735 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.748+0000 7f4bb6ffd640 1 -- 172.21.15.104:0/2878322893 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ba4004eb0 con 0x7f4bc0149dc0 2024-03-29T11:26:11.736 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.749+0000 7f4bb6ffd640 10 client.?.objecter ms_handle_connect 0x7f4bc014ff40 2024-03-29T11:26:11.736 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.749+0000 7f4bb6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:11.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.750+0000 7f4bb6ffd640 1 -- 172.21.15.104:0/2878322893 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1741470685 0 0) 0x7f4bc00ab760 con 0x7f4bc014ff40 2024-03-29T11:26:11.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.750+0000 7f4bb6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:11.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.750+0000 7f4bb6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:11.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.750+0000 7f4bb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:11.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.750+0000 7f4bb6ffd640 10 monclient(hunting): my global_id is 4462 2024-03-29T11:26:11.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.750+0000 7f4bb6ffd640 1 -- 172.21.15.104:0/2878322893 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bc00ab430 con 0x7f4bc014ff40 2024-03-29T11:26:11.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.751+0000 7f4bb6ffd640 1 -- 172.21.15.104:0/2878322893 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1780924187 0 0) 0x7f4ba4004eb0 con 0x7f4bc0149dc0 2024-03-29T11:26:11.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.751+0000 7f4bb6ffd640 1 -- 172.21.15.104:0/2878322893 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4bc00ab760 con 0x7f4bc0149dc0 2024-03-29T11:26:11.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.751+0000 7f4bb6ffd640 1 -- 172.21.15.104:0/2878322893 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f4bbc001830 con 0x7f4bc0149dc0 2024-03-29T11:26:11.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.751+0000 7f4bb6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:11.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.751+0000 7f4bb6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:11.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.751+0000 7f4bb6ffd640 10 monclient(hunting): dump: 2024-03-29T11:26:11.740 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:26:11.741 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:26:11.741 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:11.741 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:11.741 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:11.741 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:26:11.741 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:26:11.741 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:26:11.741 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:26:11.741 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:26:11.741 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.751+0000 7f4bb6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:11.741 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.751+0000 7f4bb6ffd640 1 -- 172.21.15.104:0/2878322893 <== mon.2 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3280632435 0 0) 0x7f4bc00ab760 con 0x7f4bc0149dc0 2024-03-29T11:26:11.741 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.751+0000 7f4bb6ffd640 1 -- 172.21.15.104:0/2878322893 >> v1:172.21.15.119:6789/0 conn(0x7f4bc014ff40 legacy=0x7f4bc00abc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:11.741 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.751+0000 7f4bb6ffd640 1 -- 172.21.15.104:0/2878322893 >> v1:172.21.15.104:6789/0 conn(0x7f4bc00a9930 legacy=0x7f4bc013f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:11.741 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.751+0000 7f4bb6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:26:11.741 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.751+0000 7f4bb6ffd640 1 monclient: found mon.c 2024-03-29T11:26:11.742 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.751+0000 7f4bb6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:11.742 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.751+0000 7f4bb6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:26:11.742 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.751+0000 7f4bb6ffd640 1 -- 172.21.15.104:0/2878322893 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc013e920 con 0x7f4bc0149dc0 2024-03-29T11:26:11.742 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.751+0000 7f4bb6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:26:11.742 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.751+0000 7f4bb6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:26:11.742 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.751+0000 7f4bb6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:11.742 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.752+0000 7f4bc85ba640 5 monclient: authenticate success, global_id 4493 2024-03-29T11:26:11.742 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.752+0000 7f4bc85ba640 10 monclient: _renew_subs 2024-03-29T11:26:11.742 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.752+0000 7f4bc85ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:26:11.742 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.752+0000 7f4bc85ba640 1 -- 172.21.15.104:0/2878322893 --> v1:172.21.15.104:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bc013ebd0 con 0x7f4bc0149dc0 2024-03-29T11:26:11.742 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.752+0000 7f4bc85ba640 10 client.4493.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:11.742 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.752+0000 7f4bc85ba640 10 monclient: _renew_subs 2024-03-29T11:26:11.742 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.752+0000 7f4bc85ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:26:11.742 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.752+0000 7f4bc85ba640 1 -- 172.21.15.104:0/2878322893 --> v1:172.21.15.104:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bc0072380 con 0x7f4bc0149dc0 2024-03-29T11:26:11.742 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.752+0000 7f4bc85ba640 1 librados: init done 2024-03-29T11:26:11.743 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.752+0000 7f4bb6ffd640 1 -- 172.21.15.104:0/2878322893 <== mon.2 v1:172.21.15.104:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4bbc001ce0 con 0x7f4bc0149dc0 2024-03-29T11:26:11.743 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.752+0000 7f4bb6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:11.743 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.752+0000 7f4bc4b53640 4 set_mon_vals no callback set 2024-03-29T11:26:11.743 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.753+0000 7f4bb6ffd640 1 -- 172.21.15.104:0/2878322893 <== mon.2 v1:172.21.15.104:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f4bbc004980 con 0x7f4bc0149dc0 2024-03-29T11:26:11.743 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.753+0000 7f4bb6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:11.743 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.753+0000 7f4bb6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:11.743 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.753+0000 7f4bb6ffd640 10 monclient: dump: 2024-03-29T11:26:11.743 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:26:11.743 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:26:11.743 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:11.743 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:11.743 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:11.743 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:26:11.743 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:26:11.743 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:26:11.744 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:26:11.744 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:26:11.745 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.754+0000 7f4bb6ffd640 1 -- 172.21.15.104:0/2878322893 <== mon.2 v1:172.21.15.104:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f4bbc004c00 con 0x7f4bc0149dc0 2024-03-29T11:26:11.745 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.756+0000 7f4bb6ffd640 1 -- 172.21.15.104:0/2878322893 <== mon.2 v1:172.21.15.104:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f4bbc084a10 con 0x7f4bc0149dc0 2024-03-29T11:26:11.745 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.756+0000 7f4bb6ffd640 10 client.4493.objecter ms_dispatch 0x7f4bc0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:11.745 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.756+0000 7f4bb6ffd640 3 client.4493.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:11.745 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.756+0000 7f4bb6ffd640 3 client.4493.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:11.745 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.757+0000 7f4bb6ffd640 20 client.4493.objecter dump_active .. 0 homeless 2024-03-29T11:26:11.745 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.757+0000 7f4bc85ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:11.745 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.757+0000 7f4bc85ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:11.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.757+0000 7f4bc85ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:26:11.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.757+0000 7f4bc85ba640 1 -- 172.21.15.104:0/2878322893 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ba8005080 con 0x7f4bc0149dc0 2024-03-29T11:26:11.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.757+0000 7f4bc6356640 10 monclient: get_auth_request con 0x7f4ba406da50 auth_method 0 2024-03-29T11:26:11.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.757+0000 7f4bb6ffd640 10 client.4493.objecter ms_handle_connect 0x7f4ba406da50 2024-03-29T11:26:11.750 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.765+0000 7f4bb6ffd640 1 -- 172.21.15.104:0/2878322893 <== mon.2 v1:172.21.15.104:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4bbc0502f0 con 0x7f4bc0149dc0 2024-03-29T11:26:11.750 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.765+0000 7f4bb6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:11.750 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:11.765+0000 7f4bb6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:12.083 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:12.097+0000 7f4bc85ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:12.083 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:12.097+0000 7f4bc85ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:12.083 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:12.097+0000 7f4bc85ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:26:12.083 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:12.097+0000 7f4bc85ba640 1 -- 172.21.15.104:0/2878322893 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ba8005330 con 0x7f4bc0149dc0 2024-03-29T11:26:12.085 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:12.100+0000 7f4bb6ffd640 1 -- 172.21.15.104:0/2878322893 <== mon.2 v1:172.21.15.104:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f4bbc052220 con 0x7f4bc0149dc0 2024-03-29T11:26:12.085 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:12.100+0000 7f4bb6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:12.085 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:12.100+0000 7f4bb6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:12.086 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:26:12.086 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:12.089 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:12.103+0000 7f4bc85ba640 10 librados: watch_flush enter 2024-03-29T11:26:12.090 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:12.104+0000 7f4bc85ba640 10 librados: watch_flush exit 2024-03-29T11:26:12.090 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:12.104+0000 7f4bc85ba640 20 client.4493.objecter shutdown clearing up homeless session... 2024-03-29T11:26:12.090 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:12.104+0000 7f4bc85ba640 10 client.4493.objecter successfully canceled tick 2024-03-29T11:26:12.090 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:12.104+0000 7f4bc85ba640 1 -- 172.21.15.104:0/2878322893 >> v1:172.21.15.104:6816/3341126660 conn(0x7f4ba406da50 legacy=0x7f4ba406ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:12.090 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:12.104+0000 7f4bc85ba640 10 monclient: shutdown 2024-03-29T11:26:12.090 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:12.104+0000 7f4bc85ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:12.090 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:12.104+0000 7f4bc85ba640 1 -- 172.21.15.104:0/2878322893 >> v1:172.21.15.104:6790/0 conn(0x7f4bc0149dc0 legacy=0x7f4bc0071a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:12.090 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:12.104+0000 7f4bc85ba640 1 -- 172.21.15.104:0/2878322893 shutdown_connections 2024-03-29T11:26:12.090 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:12.104+0000 7f4bc85ba640 1 -- 172.21.15.104:0/2878322893 >> 172.21.15.104:0/2878322893 conn(0x7f4bc00b35e0 msgr2=0x7f4bc00b2850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:12.091 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:12.105+0000 7f4bc85ba640 1 -- 172.21.15.104:0/2878322893 shutdown_connections 2024-03-29T11:26:12.091 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:12.105+0000 7f4bc85ba640 1 -- 172.21.15.104:0/2878322893 wait complete. 2024-03-29T11:26:12.091 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:12.105+0000 7f4bc85ba640 1 librados: shutdown 2024-03-29T11:26:12.108 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:18.914 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:19.069 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.083+0000 7f695f78f640 10 monclient: get_monmap_and_config 2024-03-29T11:26:19.069 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.083+0000 7f695f78f640 10 monclient: build_initial_monmap 2024-03-29T11:26:19.069 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.083+0000 7f695f78f640 10 monclient: monmap: 2024-03-29T11:26:19.069 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:26:19.069 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:26:19.069 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:26:19.084350+0000 2024-03-29T11:26:19.069 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:26:19.084350+0000 2024-03-29T11:26:19.069 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:19.070 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:26:19.070 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:26:19.070 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:26:19.070 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:26:19.070 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:26:19.073 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.084+0000 7f695f78f640 1 Processor -- start 2024-03-29T11:26:19.073 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.084+0000 7f695f78f640 1 -- start start 2024-03-29T11:26:19.073 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.084+0000 7f695f78f640 10 monclient: init 2024-03-29T11:26:19.073 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.085+0000 7f695f78f640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:19.073 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.085+0000 7f695f78f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:26:19.073 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.085+0000 7f695f78f640 10 monclient(hunting): picked mon.noname-a con 0x7f69581457f0 addr v1:172.21.15.104:6789/0 2024-03-29T11:26:19.073 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.085+0000 7f695f78f640 10 monclient(hunting): picked mon.noname-b con 0x7f695813f650 addr v1:172.21.15.104:6790/0 2024-03-29T11:26:19.073 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.085+0000 7f695f78f640 10 monclient(hunting): picked mon.noname-c con 0x7f69580a9d70 addr v1:172.21.15.119:6789/0 2024-03-29T11:26:19.073 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.085+0000 7f695f78f640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69580aaef0 con 0x7f69581457f0 2024-03-29T11:26:19.073 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.085+0000 7f695f78f640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6958145bd0 con 0x7f69580a9d70 2024-03-29T11:26:19.074 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.085+0000 7f695f78f640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6958145eb0 con 0x7f695813f650 2024-03-29T11:26:19.074 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.085+0000 7f695f78f640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:19.074 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.085+0000 7f695f78f640 10 monclient(hunting): authenticate will time out at 1360.336308s 2024-03-29T11:26:19.074 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.085+0000 7f695dd2c640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f69580a9d70 0x7f695814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:49990/0 (socket says 172.21.15.104:49990) 2024-03-29T11:26:19.074 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.085+0000 7f695dd2c640 1 -- 172.21.15.104:0/2346416284 learned_addr learned my addr 172.21.15.104:0/2346416284 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:26:19.074 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.086+0000 7f694bfff640 1 -- 172.21.15.104:0/2346416284 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1727532922 0 0) 0x7f6958145bd0 con 0x7f69580a9d70 2024-03-29T11:26:19.074 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.086+0000 7f694bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:19.074 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.086+0000 7f694bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:19.074 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.086+0000 7f694bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:19.074 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.086+0000 7f694bfff640 10 monclient(hunting): my global_id is 4465 2024-03-29T11:26:19.074 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.086+0000 7f694bfff640 1 -- 172.21.15.104:0/2346416284 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6944003520 con 0x7f69580a9d70 2024-03-29T11:26:19.074 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.086+0000 7f694bfff640 1 -- 172.21.15.104:0/2346416284 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1326164113 0 0) 0x7f6944003520 con 0x7f69580a9d70 2024-03-29T11:26:19.074 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.086+0000 7f694bfff640 1 -- 172.21.15.104:0/2346416284 >> v1:172.21.15.104:6790/0 conn(0x7f695813f650 legacy=0x7f69580b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:19.075 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.088+0000 7f694bfff640 1 -- 172.21.15.104:0/2346416284 >> v1:172.21.15.104:6789/0 conn(0x7f69581457f0 legacy=0x7f695813ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:19.075 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.089+0000 7f694bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:26:19.076 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.089+0000 7f694bfff640 1 monclient: found mon.noname-c 2024-03-29T11:26:19.076 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.089+0000 7f694bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:19.076 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.089+0000 7f694bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.119:6789/0 2024-03-29T11:26:19.076 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.089+0000 7f694bfff640 1 -- 172.21.15.104:0/2346416284 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6958146340 con 0x7f69580a9d70 2024-03-29T11:26:19.076 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.089+0000 7f694bfff640 10 monclient: _finish_auth 0 2024-03-29T11:26:19.076 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.089+0000 7f694bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:26:19.076 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.089+0000 7f694bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:19.076 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.089+0000 7f694bfff640 1 -- 172.21.15.104:0/2346416284 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f6950001a10 con 0x7f69580a9d70 2024-03-29T11:26:19.076 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.089+0000 7f694bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:19.076 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.089+0000 7f694bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:19.076 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.089+0000 7f694bfff640 10 monclient: dump: 2024-03-29T11:26:19.076 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:26:19.077 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:26:19.077 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:19.077 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:19.077 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:19.077 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:26:19.077 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:26:19.077 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:26:19.077 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:26:19.077 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:26:19.077 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.089+0000 7f695f78f640 5 monclient: authenticate success, global_id 4465 2024-03-29T11:26:19.077 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.089+0000 7f695f78f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:19.078 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.089+0000 7f694bfff640 1 -- 172.21.15.104:0/2346416284 <== mon.1 v1:172.21.15.119:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6950001be0 con 0x7f69580a9d70 2024-03-29T11:26:19.078 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.089+0000 7f694bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:19.078 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.089+0000 7f694bfff640 1 -- 172.21.15.104:0/2346416284 <== mon.1 v1:172.21.15.119:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f6950002030 con 0x7f69580a9d70 2024-03-29T11:26:19.078 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.089+0000 7f694bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:19.078 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.089+0000 7f694bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:19.078 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.090+0000 7f694bfff640 10 monclient: dump: 2024-03-29T11:26:19.078 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:26:19.078 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:26:19.078 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:19.078 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:19.078 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:19.078 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:26:19.078 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:26:19.079 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:26:19.079 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:26:19.079 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:26:19.079 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.090+0000 7f695f78f640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:19.079 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.090+0000 7f695f78f640 4 set_mon_vals no callback set 2024-03-29T11:26:19.081 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.090+0000 7f695f78f640 10 monclient: shutdown 2024-03-29T11:26:19.081 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.090+0000 7f695f78f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:19.081 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.090+0000 7f695f78f640 1 -- 172.21.15.104:0/2346416284 >> v1:172.21.15.119:6789/0 conn(0x7f69580a9d70 legacy=0x7f695814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:19.081 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.091+0000 7f695f78f640 1 -- 172.21.15.104:0/2346416284 shutdown_connections 2024-03-29T11:26:19.081 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.091+0000 7f695f78f640 1 -- 172.21.15.104:0/2346416284 >> 172.21.15.104:0/2346416284 conn(0x7f6958066a00 msgr2=0x7f6958066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:19.081 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.091+0000 7f695f78f640 1 -- 172.21.15.104:0/2346416284 shutdown_connections 2024-03-29T11:26:19.081 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.091+0000 7f695f78f640 1 -- 172.21.15.104:0/2346416284 wait complete. 2024-03-29T11:26:19.082 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.092+0000 7f695f78f640 10 monclient: build_initial_monmap 2024-03-29T11:26:19.082 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.092+0000 7f695f78f640 10 monclient: monmap: 2024-03-29T11:26:19.082 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:26:19.082 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:19.082 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:26:19.082 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:26:19.082 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:19.082 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:26:19.082 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:26:19.082 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:26:19.082 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:26:19.082 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:26:19.082 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.092+0000 7f695f78f640 1 librados: starting msgr at 2024-03-29T11:26:19.083 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.092+0000 7f695f78f640 1 librados: starting objecter 2024-03-29T11:26:19.083 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.093+0000 7f695f78f640 1 Processor -- start 2024-03-29T11:26:19.083 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.093+0000 7f695f78f640 1 -- start start 2024-03-29T11:26:19.083 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.093+0000 7f695f78f640 1 librados: setting wanted keys 2024-03-29T11:26:19.083 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.093+0000 7f695f78f640 1 librados: calling monclient init 2024-03-29T11:26:19.083 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.093+0000 7f695f78f640 10 monclient: init 2024-03-29T11:26:19.083 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.093+0000 7f695f78f640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:19.083 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.093+0000 7f695f78f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:26:19.083 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.093+0000 7f695f78f640 10 monclient(hunting): picked mon.noname-b con 0x7f69580a9d70 addr v1:172.21.15.119:6789/0 2024-03-29T11:26:19.084 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.093+0000 7f695f78f640 10 monclient(hunting): picked mon.noname-a con 0x7f695813f650 addr v1:172.21.15.104:6789/0 2024-03-29T11:26:19.084 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.093+0000 7f695f78f640 10 monclient(hunting): picked mon.noname-c con 0x7f69581457f0 addr v1:172.21.15.104:6790/0 2024-03-29T11:26:19.084 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.093+0000 7f695f78f640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69581dfc10 con 0x7f695813f650 2024-03-29T11:26:19.084 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.093+0000 7f695f78f640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6958070060 con 0x7f69580a9d70 2024-03-29T11:26:19.084 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.093+0000 7f695f78f640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69580718e0 con 0x7f69581457f0 2024-03-29T11:26:19.084 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.093+0000 7f695f78f640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:19.084 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.093+0000 7f695f78f640 10 monclient(hunting): authenticate will time out at 1360.344974s 2024-03-29T11:26:19.085 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.097+0000 7f695dd2c640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f69581457f0 0x7f69581df500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:37865/0 (socket says 172.21.15.104:37865) 2024-03-29T11:26:19.085 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.097+0000 7f695dd2c640 1 -- 172.21.15.104:0/534733004 learned_addr learned my addr 172.21.15.104:0/534733004 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:26:19.085 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.097+0000 7f6949ffb640 10 client.?.objecter ms_handle_connect 0x7f69581457f0 2024-03-29T11:26:19.086 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.097+0000 7f6949ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:19.086 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.097+0000 7f6949ffb640 10 client.?.objecter ms_handle_connect 0x7f69580a9d70 2024-03-29T11:26:19.086 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.097+0000 7f6949ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:19.086 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.097+0000 7f6949ffb640 10 client.?.objecter ms_handle_connect 0x7f695813f650 2024-03-29T11:26:19.086 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.097+0000 7f6949ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:19.086 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.097+0000 7f6949ffb640 1 -- 172.21.15.104:0/534733004 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1004336913 0 0) 0x7f69580718e0 con 0x7f69581457f0 2024-03-29T11:26:19.086 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.097+0000 7f6949ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:19.086 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.097+0000 7f6949ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:19.086 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.097+0000 7f6949ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:19.086 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.097+0000 7f6949ffb640 10 monclient(hunting): my global_id is 4496 2024-03-29T11:26:19.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.097+0000 7f6949ffb640 1 -- 172.21.15.104:0/534733004 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f693c004eb0 con 0x7f69581457f0 2024-03-29T11:26:19.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.098+0000 7f6949ffb640 1 -- 172.21.15.104:0/534733004 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1043051987 0 0) 0x7f6958070060 con 0x7f69580a9d70 2024-03-29T11:26:19.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.098+0000 7f6949ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:19.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.098+0000 7f6949ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:19.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.098+0000 7f6949ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:19.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.098+0000 7f6949ffb640 10 monclient(hunting): my global_id is 4468 2024-03-29T11:26:19.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.098+0000 7f6949ffb640 1 -- 172.21.15.104:0/534733004 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69580718e0 con 0x7f69580a9d70 2024-03-29T11:26:19.088 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.099+0000 7f6949ffb640 1 -- 172.21.15.104:0/534733004 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2481053482 0 0) 0x7f69581dfc10 con 0x7f695813f650 2024-03-29T11:26:19.088 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.099+0000 7f6949ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:19.088 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.099+0000 7f6949ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:19.088 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.099+0000 7f6949ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:19.088 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.099+0000 7f6949ffb640 10 monclient(hunting): my global_id is 4488 2024-03-29T11:26:19.089 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.099+0000 7f6949ffb640 1 -- 172.21.15.104:0/534733004 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6958070060 con 0x7f695813f650 2024-03-29T11:26:19.089 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.099+0000 7f6949ffb640 1 -- 172.21.15.104:0/534733004 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2581487805 0 0) 0x7f693c004eb0 con 0x7f69581457f0 2024-03-29T11:26:19.089 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.099+0000 7f6949ffb640 1 -- 172.21.15.104:0/534733004 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f69581dfc10 con 0x7f69581457f0 2024-03-29T11:26:19.089 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.099+0000 7f6949ffb640 1 -- 172.21.15.104:0/534733004 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 38465298 0 0) 0x7f69580718e0 con 0x7f69580a9d70 2024-03-29T11:26:19.089 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.099+0000 7f6949ffb640 1 -- 172.21.15.104:0/534733004 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f693c004eb0 con 0x7f69580a9d70 2024-03-29T11:26:19.089 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.099+0000 7f6949ffb640 1 -- 172.21.15.104:0/534733004 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f6950001ef0 con 0x7f69581457f0 2024-03-29T11:26:19.089 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.099+0000 7f6949ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:19.089 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.099+0000 7f6949ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:19.089 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.099+0000 7f6949ffb640 10 monclient(hunting): dump: 2024-03-29T11:26:19.089 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:26:19.089 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:26:19.089 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:19.089 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:19.090 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:19.090 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:26:19.090 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:26:19.090 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:26:19.090 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:26:19.090 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:26:19.090 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.099+0000 7f6949ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:19.090 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.099+0000 7f6949ffb640 1 -- 172.21.15.104:0/534733004 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f69540016f0 con 0x7f69580a9d70 2024-03-29T11:26:19.090 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.099+0000 7f6949ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:19.090 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.099+0000 7f6949ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:19.090 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.099+0000 7f6949ffb640 10 monclient(hunting): dump: 2024-03-29T11:26:19.090 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:26:19.091 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:26:19.091 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:19.091 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:19.091 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:19.091 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:26:19.091 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:26:19.091 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:26:19.091 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:26:19.091 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:26:19.091 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.099+0000 7f6949ffb640 1 -- 172.21.15.104:0/534733004 <== mon.2 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1371758010 0 0) 0x7f69581dfc10 con 0x7f69581457f0 2024-03-29T11:26:19.091 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.099+0000 7f6949ffb640 1 -- 172.21.15.104:0/534733004 >> v1:172.21.15.119:6789/0 conn(0x7f69580a9d70 legacy=0x7f69580705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:19.091 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.099+0000 7f6949ffb640 1 -- 172.21.15.104:0/534733004 >> v1:172.21.15.104:6789/0 conn(0x7f695813f650 legacy=0x7f69581dedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:19.091 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.099+0000 7f6949ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:26:19.091 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.099+0000 7f6949ffb640 1 monclient: found mon.c 2024-03-29T11:26:19.091 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.099+0000 7f6949ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:19.092 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.099+0000 7f6949ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:26:19.092 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.099+0000 7f6949ffb640 1 -- 172.21.15.104:0/534733004 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6958071bc0 con 0x7f69581457f0 2024-03-29T11:26:19.092 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.100+0000 7f6949ffb640 10 monclient: _finish_auth 0 2024-03-29T11:26:19.092 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.100+0000 7f6949ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:26:19.092 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.100+0000 7f6949ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:19.092 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.100+0000 7f6949ffb640 1 -- 172.21.15.104:0/534733004 <== mon.2 v1:172.21.15.104:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6950002050 con 0x7f69581457f0 2024-03-29T11:26:19.092 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.100+0000 7f6949ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:19.092 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.100+0000 7f6949ffb640 1 -- 172.21.15.104:0/534733004 <== mon.2 v1:172.21.15.104:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f6950005110 con 0x7f69581457f0 2024-03-29T11:26:19.092 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.100+0000 7f6949ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:19.092 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.100+0000 7f6949ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:19.092 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.100+0000 7f6949ffb640 10 monclient: dump: 2024-03-29T11:26:19.093 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:26:19.093 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:26:19.093 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:19.093 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:19.093 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:19.093 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:26:19.093 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:26:19.093 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:26:19.093 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:26:19.093 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:26:19.093 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.101+0000 7f694affd640 4 set_mon_vals no callback set 2024-03-29T11:26:19.093 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.101+0000 7f695f78f640 5 monclient: authenticate success, global_id 4496 2024-03-29T11:26:19.093 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.101+0000 7f695f78f640 10 monclient: _renew_subs 2024-03-29T11:26:19.093 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.101+0000 7f695f78f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:26:19.094 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.101+0000 7f695f78f640 1 -- 172.21.15.104:0/534733004 --> v1:172.21.15.104:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6958071ea0 con 0x7f69581457f0 2024-03-29T11:26:19.094 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.101+0000 7f695f78f640 10 client.4496.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:19.094 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.101+0000 7f695f78f640 10 monclient: _renew_subs 2024-03-29T11:26:19.094 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.101+0000 7f695f78f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:26:19.094 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.101+0000 7f695f78f640 1 -- 172.21.15.104:0/534733004 --> v1:172.21.15.104:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6958136520 con 0x7f69581457f0 2024-03-29T11:26:19.094 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.101+0000 7f695f78f640 1 librados: init done 2024-03-29T11:26:19.095 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.103+0000 7f695f78f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:19.095 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.103+0000 7f695f78f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:19.095 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.103+0000 7f695f78f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:26:19.095 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.103+0000 7f695f78f640 1 -- 172.21.15.104:0/534733004 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6958075f10 con 0x7f69581457f0 2024-03-29T11:26:19.095 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.103+0000 7f6949ffb640 1 -- 172.21.15.104:0/534733004 <== mon.2 v1:172.21.15.104:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f695001b0e0 con 0x7f69581457f0 2024-03-29T11:26:19.095 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.104+0000 7f6949ffb640 1 -- 172.21.15.104:0/534733004 <== mon.2 v1:172.21.15.104:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f6950084e40 con 0x7f69581457f0 2024-03-29T11:26:19.095 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.105+0000 7f6949ffb640 10 client.4496.objecter ms_dispatch 0x7f6958000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:19.095 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.105+0000 7f695d52b640 10 monclient: get_auth_request con 0x7f693c07f060 auth_method 0 2024-03-29T11:26:19.095 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.105+0000 7f6949ffb640 3 client.4496.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:19.096 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.105+0000 7f6949ffb640 3 client.4496.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:19.096 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.105+0000 7f6949ffb640 20 client.4496.objecter dump_active .. 0 homeless 2024-03-29T11:26:19.096 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.105+0000 7f6949ffb640 10 client.4496.objecter ms_handle_connect 0x7f693c07f060 2024-03-29T11:26:19.097 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.111+0000 7f6949ffb640 1 -- 172.21.15.104:0/534733004 <== mon.2 v1:172.21.15.104:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6950050720 con 0x7f69581457f0 2024-03-29T11:26:19.097 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.111+0000 7f6949ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:19.097 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.111+0000 7f6949ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:19.420 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.434+0000 7f695f78f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:19.420 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.434+0000 7f695f78f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:19.420 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.434+0000 7f695f78f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:26:19.420 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.434+0000 7f695f78f640 1 -- 172.21.15.104:0/534733004 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f69580b3680 con 0x7f69581457f0 2024-03-29T11:26:19.421 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.435+0000 7f6949ffb640 1 -- 172.21.15.104:0/534733004 <== mon.2 v1:172.21.15.104:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f6950052650 con 0x7f69581457f0 2024-03-29T11:26:19.421 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.435+0000 7f6949ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:19.421 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.435+0000 7f6949ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:19.421 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:26:19.422 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:19.424 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.439+0000 7f695f78f640 10 librados: watch_flush enter 2024-03-29T11:26:19.425 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.439+0000 7f695f78f640 10 librados: watch_flush exit 2024-03-29T11:26:19.425 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.439+0000 7f695f78f640 20 client.4496.objecter shutdown clearing up homeless session... 2024-03-29T11:26:19.425 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.439+0000 7f695f78f640 10 client.4496.objecter successfully canceled tick 2024-03-29T11:26:19.427 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.441+0000 7f695f78f640 1 -- 172.21.15.104:0/534733004 >> v1:172.21.15.104:6816/3341126660 conn(0x7f693c07f060 legacy=0x7f693c081520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:19.427 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.441+0000 7f695f78f640 10 monclient: shutdown 2024-03-29T11:26:19.427 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.441+0000 7f695f78f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:19.427 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.441+0000 7f695f78f640 1 -- 172.21.15.104:0/534733004 >> v1:172.21.15.104:6790/0 conn(0x7f69581457f0 legacy=0x7f69581df500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:19.427 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.441+0000 7f695f78f640 1 -- 172.21.15.104:0/534733004 shutdown_connections 2024-03-29T11:26:19.427 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.441+0000 7f695f78f640 1 -- 172.21.15.104:0/534733004 >> 172.21.15.104:0/534733004 conn(0x7f6958066a00 msgr2=0x7f69580b60a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:19.427 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.441+0000 7f695f78f640 1 -- 172.21.15.104:0/534733004 shutdown_connections 2024-03-29T11:26:19.429 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.441+0000 7f695f78f640 1 -- 172.21.15.104:0/534733004 wait complete. 2024-03-29T11:26:19.429 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:19.441+0000 7f695f78f640 1 librados: shutdown 2024-03-29T11:26:19.444 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:26.249 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:26.403 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.417+0000 7f6dfe231640 10 monclient: get_monmap_and_config 2024-03-29T11:26:26.403 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.417+0000 7f6dfe231640 10 monclient: build_initial_monmap 2024-03-29T11:26:26.403 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.417+0000 7f6dfe231640 10 monclient: monmap: 2024-03-29T11:26:26.403 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:26:26.403 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:26:26.403 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:26:26.418088+0000 2024-03-29T11:26:26.403 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:26:26.418088+0000 2024-03-29T11:26:26.403 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:26.404 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:26:26.404 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:26:26.404 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:26:26.404 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:26:26.404 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:26:26.404 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.418+0000 7f6dfe231640 1 Processor -- start 2024-03-29T11:26:26.405 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.419+0000 7f6dfe231640 1 -- start start 2024-03-29T11:26:26.405 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.419+0000 7f6dfe231640 10 monclient: init 2024-03-29T11:26:26.410 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.420+0000 7f6dfe231640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:26.410 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.420+0000 7f6dfe231640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:26:26.410 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.420+0000 7f6dfe231640 10 monclient(hunting): picked mon.noname-b con 0x7f6df806ab70 addr v1:172.21.15.104:6790/0 2024-03-29T11:26:26.410 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.420+0000 7f6dfe231640 10 monclient(hunting): picked mon.noname-c con 0x7f6df806af50 addr v1:172.21.15.119:6789/0 2024-03-29T11:26:26.410 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.420+0000 7f6dfe231640 10 monclient(hunting): picked mon.noname-a con 0x7f6df8141fb0 addr v1:172.21.15.104:6789/0 2024-03-29T11:26:26.410 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.420+0000 7f6dfe231640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6df80b0d50 con 0x7f6df8141fb0 2024-03-29T11:26:26.410 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.420+0000 7f6dfe231640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6df80b1030 con 0x7f6df806af50 2024-03-29T11:26:26.410 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.420+0000 7f6dfe231640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6df80afa70 con 0x7f6df806ab70 2024-03-29T11:26:26.410 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.420+0000 7f6dfe231640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:26.410 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.420+0000 7f6dfe231640 10 monclient(hunting): authenticate will time out at 1367.671440s 2024-03-29T11:26:26.411 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.424+0000 7f6df7fff640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f6df8141fb0 0x7f6df80715b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:45533/0 (socket says 172.21.15.104:45533) 2024-03-29T11:26:26.411 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.424+0000 7f6df7fff640 1 -- 172.21.15.104:0/2821220725 learned_addr learned my addr 172.21.15.104:0/2821220725 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:26:26.412 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.426+0000 7f6df67fc640 1 -- 172.21.15.104:0/2821220725 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4070668482 0 0) 0x7f6df80b0d50 con 0x7f6df8141fb0 2024-03-29T11:26:26.412 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.426+0000 7f6df67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:26:26.412 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.426+0000 7f6df67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:26.412 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.426+0000 7f6df67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:26.412 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.426+0000 7f6df67fc640 10 monclient(hunting): my global_id is 4491 2024-03-29T11:26:26.412 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.426+0000 7f6df67fc640 1 -- 172.21.15.104:0/2821220725 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6de0003520 con 0x7f6df8141fb0 2024-03-29T11:26:26.416 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.427+0000 7f6df67fc640 1 -- 172.21.15.104:0/2821220725 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1542016560 0 0) 0x7f6de0003520 con 0x7f6df8141fb0 2024-03-29T11:26:26.416 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.427+0000 7f6df67fc640 1 -- 172.21.15.104:0/2821220725 >> v1:172.21.15.104:6790/0 conn(0x7f6df806ab70 legacy=0x7f6df80af2a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:26.416 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.427+0000 7f6df67fc640 1 -- 172.21.15.104:0/2821220725 >> v1:172.21.15.119:6789/0 conn(0x7f6df806af50 legacy=0x7f6df8072290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:26:26.416 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.427+0000 7f6df67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:26:26.416 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.427+0000 7f6df67fc640 1 monclient: found mon.noname-a 2024-03-29T11:26:26.416 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.427+0000 7f6df67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:26.416 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.427+0000 7f6df67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.104:6789/0 2024-03-29T11:26:26.416 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.428+0000 7f6df67fc640 1 -- 172.21.15.104:0/2821220725 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6df81423e0 con 0x7f6df8141fb0 2024-03-29T11:26:26.416 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.428+0000 7f6df67fc640 10 monclient: _finish_auth 0 2024-03-29T11:26:26.416 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.428+0000 7f6df67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:26:26.416 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.428+0000 7f6df67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:26.416 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.428+0000 7f6df67fc640 1 -- 172.21.15.104:0/2821220725 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f6df0001a10 con 0x7f6df8141fb0 2024-03-29T11:26:26.416 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.428+0000 7f6df67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:26.417 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.428+0000 7f6df67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:26.417 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.428+0000 7f6df67fc640 10 monclient: dump: 2024-03-29T11:26:26.417 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:26:26.417 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:26:26.417 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:26.417 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:26.417 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:26.417 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:26:26.417 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:26:26.417 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:26:26.417 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:26:26.417 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:26:26.417 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.428+0000 7f6df67fc640 1 -- 172.21.15.104:0/2821220725 <== mon.0 v1:172.21.15.104:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6df0001ca0 con 0x7f6df8141fb0 2024-03-29T11:26:26.417 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.428+0000 7f6df67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:26.418 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.428+0000 7f6df67fc640 1 -- 172.21.15.104:0/2821220725 <== mon.0 v1:172.21.15.104:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f6df0002030 con 0x7f6df8141fb0 2024-03-29T11:26:26.418 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.428+0000 7f6df67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:26.418 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.428+0000 7f6df67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:26.418 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.428+0000 7f6df67fc640 10 monclient: dump: 2024-03-29T11:26:26.418 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:26:26.418 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:26:26.418 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:26.418 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:26.418 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:26.418 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:26:26.418 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:26:26.418 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:26:26.419 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:26:26.419 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:26:26.419 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.428+0000 7f6dfe231640 5 monclient: authenticate success, global_id 4491 2024-03-29T11:26:26.419 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.428+0000 7f6dfe231640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:26.419 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.428+0000 7f6dfe231640 4 set_mon_vals no callback set 2024-03-29T11:26:26.419 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.429+0000 7f6dfe231640 10 monclient: shutdown 2024-03-29T11:26:26.419 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.429+0000 7f6dfe231640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:26.419 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.429+0000 7f6dfe231640 1 -- 172.21.15.104:0/2821220725 >> v1:172.21.15.104:6789/0 conn(0x7f6df8141fb0 legacy=0x7f6df80715b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:26.420 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.429+0000 7f6dfe231640 1 -- 172.21.15.104:0/2821220725 shutdown_connections 2024-03-29T11:26:26.420 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.429+0000 7f6dfe231640 1 -- 172.21.15.104:0/2821220725 >> 172.21.15.104:0/2821220725 conn(0x7f6df8067470 msgr2=0x7f6df80b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:26.421 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.430+0000 7f6dfe231640 1 -- 172.21.15.104:0/2821220725 shutdown_connections 2024-03-29T11:26:26.422 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.430+0000 7f6dfe231640 1 -- 172.21.15.104:0/2821220725 wait complete. 2024-03-29T11:26:26.422 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.431+0000 7f6dfe231640 10 monclient: build_initial_monmap 2024-03-29T11:26:26.422 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.431+0000 7f6dfe231640 10 monclient: monmap: 2024-03-29T11:26:26.422 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:26:26.422 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:26.422 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:26:26.422 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:26:26.422 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:26.422 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:26:26.422 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:26:26.422 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:26:26.422 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:26:26.422 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:26:26.423 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.432+0000 7f6dfe231640 1 librados: starting msgr at 2024-03-29T11:26:26.423 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.432+0000 7f6dfe231640 1 librados: starting objecter 2024-03-29T11:26:26.423 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.432+0000 7f6dfe231640 1 Processor -- start 2024-03-29T11:26:26.423 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.432+0000 7f6dfe231640 1 -- start start 2024-03-29T11:26:26.423 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.432+0000 7f6dfe231640 1 librados: setting wanted keys 2024-03-29T11:26:26.423 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.432+0000 7f6dfe231640 1 librados: calling monclient init 2024-03-29T11:26:26.423 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.432+0000 7f6dfe231640 10 monclient: init 2024-03-29T11:26:26.423 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.433+0000 7f6dfe231640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:26.423 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.433+0000 7f6dfe231640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:26:26.423 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.433+0000 7f6dfe231640 10 monclient(hunting): picked mon.noname-a con 0x7f6df806ab70 addr v1:172.21.15.104:6789/0 2024-03-29T11:26:26.423 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.433+0000 7f6dfe231640 10 monclient(hunting): picked mon.noname-b con 0x7f6df8141fb0 addr v1:172.21.15.119:6789/0 2024-03-29T11:26:26.423 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.433+0000 7f6dfe231640 10 monclient(hunting): picked mon.noname-c con 0x7f6df80a94e0 addr v1:172.21.15.104:6790/0 2024-03-29T11:26:26.423 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.433+0000 7f6dfe231640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6df8135440 con 0x7f6df806ab70 2024-03-29T11:26:26.423 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.433+0000 7f6dfe231640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6df80ab3f0 con 0x7f6df8141fb0 2024-03-29T11:26:26.423 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.433+0000 7f6dfe231640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6df80ab6d0 con 0x7f6df80a94e0 2024-03-29T11:26:26.424 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.433+0000 7f6dfe231640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:26.424 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.433+0000 7f6dfe231640 10 monclient(hunting): authenticate will time out at 1367.684619s 2024-03-29T11:26:26.425 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.439+0000 7f6df7fff640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f6df80a94e0 0x7f6df81ee7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:33901/0 (socket says 172.21.15.104:33901) 2024-03-29T11:26:26.426 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.440+0000 7f6df6ffd640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f6df8141fb0 0x7f6df81e79b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:55977/0 (socket says 172.21.15.104:55977) 2024-03-29T11:26:26.426 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.440+0000 7f6df6ffd640 1 -- 172.21.15.104:0/3452494388 learned_addr learned my addr 172.21.15.104:0/3452494388 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:26:26.429 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.441+0000 7f6ddffff640 10 client.?.objecter ms_handle_connect 0x7f6df8141fb0 2024-03-29T11:26:26.429 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.441+0000 7f6ddffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:26.429 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.441+0000 7f6ddffff640 10 client.?.objecter ms_handle_connect 0x7f6df80a94e0 2024-03-29T11:26:26.429 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.441+0000 7f6ddffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:26.429 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.441+0000 7f6ddffff640 1 -- 172.21.15.104:0/3452494388 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1922449862 0 0) 0x7f6df80ab6d0 con 0x7f6df80a94e0 2024-03-29T11:26:26.429 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.441+0000 7f6ddffff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:26.429 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.441+0000 7f6ddffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:26.429 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.441+0000 7f6ddffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:26.429 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.441+0000 7f6ddffff640 10 monclient(hunting): my global_id is 4499 2024-03-29T11:26:26.429 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.441+0000 7f6ddffff640 1 -- 172.21.15.104:0/3452494388 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6de4004eb0 con 0x7f6df80a94e0 2024-03-29T11:26:26.430 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.441+0000 7f6ddffff640 1 -- 172.21.15.104:0/3452494388 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3717935537 0 0) 0x7f6df80ab3f0 con 0x7f6df8141fb0 2024-03-29T11:26:26.430 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.441+0000 7f6ddffff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:26.430 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.441+0000 7f6ddffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:26.430 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.441+0000 7f6ddffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:26.430 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.441+0000 7f6ddffff640 10 monclient(hunting): my global_id is 4471 2024-03-29T11:26:26.430 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.441+0000 7f6ddffff640 1 -- 172.21.15.104:0/3452494388 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6df80ab6d0 con 0x7f6df8141fb0 2024-03-29T11:26:26.430 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.442+0000 7f6ddffff640 1 -- 172.21.15.104:0/3452494388 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 521517677 0 0) 0x7f6df80ab6d0 con 0x7f6df8141fb0 2024-03-29T11:26:26.430 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.442+0000 7f6ddffff640 1 -- 172.21.15.104:0/3452494388 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6df80ab3f0 con 0x7f6df8141fb0 2024-03-29T11:26:26.430 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.442+0000 7f6ddffff640 1 -- 172.21.15.104:0/3452494388 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f6dec001920 con 0x7f6df8141fb0 2024-03-29T11:26:26.430 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.442+0000 7f6ddffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:26.430 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.442+0000 7f6ddffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:26.430 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.442+0000 7f6ddffff640 10 monclient(hunting): dump: 2024-03-29T11:26:26.430 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:26:26.431 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:26:26.431 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:26.431 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:26.431 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:26.431 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:26:26.431 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:26:26.431 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:26:26.431 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:26:26.431 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:26:26.431 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.442+0000 7f6ddffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:26.431 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.442+0000 7f6ddffff640 10 client.?.objecter ms_handle_connect 0x7f6df806ab70 2024-03-29T11:26:26.431 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.442+0000 7f6ddffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:26.431 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.442+0000 7f6ddffff640 1 -- 172.21.15.104:0/3452494388 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3975669158 0 0) 0x7f6de4004eb0 con 0x7f6df80a94e0 2024-03-29T11:26:26.431 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.442+0000 7f6ddffff640 1 -- 172.21.15.104:0/3452494388 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6df80ab6d0 con 0x7f6df80a94e0 2024-03-29T11:26:26.432 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.442+0000 7f6ddffff640 1 -- 172.21.15.104:0/3452494388 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 280961851 0 0) 0x7f6df80ab3f0 con 0x7f6df8141fb0 2024-03-29T11:26:26.432 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.442+0000 7f6ddffff640 1 -- 172.21.15.104:0/3452494388 >> v1:172.21.15.104:6790/0 conn(0x7f6df80a94e0 legacy=0x7f6df81ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:26.432 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.442+0000 7f6ddffff640 1 -- 172.21.15.104:0/3452494388 >> v1:172.21.15.104:6789/0 conn(0x7f6df806ab70 legacy=0x7f6df81e72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:26.432 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.442+0000 7f6ddffff640 10 monclient: _finish_hunting 0 2024-03-29T11:26:26.432 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.442+0000 7f6ddffff640 1 monclient: found mon.b 2024-03-29T11:26:26.432 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.443+0000 7f6ddffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:26.432 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.443+0000 7f6ddffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:26:26.432 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.443+0000 7f6ddffff640 1 -- 172.21.15.104:0/3452494388 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6df80ab9b0 con 0x7f6df8141fb0 2024-03-29T11:26:26.432 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.443+0000 7f6ddffff640 10 monclient: _finish_auth 0 2024-03-29T11:26:26.432 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.443+0000 7f6ddffff640 10 monclient: _check_auth_tickets 2024-03-29T11:26:26.432 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.443+0000 7f6ddffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:26.432 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.443+0000 7f6dfe231640 5 monclient: authenticate success, global_id 4471 2024-03-29T11:26:26.432 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.443+0000 7f6dfe231640 10 monclient: _renew_subs 2024-03-29T11:26:26.432 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.443+0000 7f6dfe231640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:26:26.432 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.443+0000 7f6dfe231640 1 -- 172.21.15.104:0/3452494388 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6df8141820 con 0x7f6df8141fb0 2024-03-29T11:26:26.433 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.443+0000 7f6dfe231640 10 client.4471.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:26.433 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.443+0000 7f6dfe231640 10 monclient: _renew_subs 2024-03-29T11:26:26.433 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.443+0000 7f6dfe231640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:26:26.433 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.443+0000 7f6dfe231640 1 -- 172.21.15.104:0/3452494388 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6df81eff00 con 0x7f6df8141fb0 2024-03-29T11:26:26.433 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.443+0000 7f6dfe231640 1 librados: init done 2024-03-29T11:26:26.435 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.444+0000 7f6ddffff640 1 -- 172.21.15.104:0/3452494388 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6dec0035f0 con 0x7f6df8141fb0 2024-03-29T11:26:26.435 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.444+0000 7f6ddffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:26.435 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.445+0000 7f6ddffff640 1 -- 172.21.15.104:0/3452494388 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f6dec003940 con 0x7f6df8141fb0 2024-03-29T11:26:26.435 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.445+0000 7f6ddffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:26.435 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.445+0000 7f6ddffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:26.435 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.445+0000 7f6ddffff640 10 monclient: dump: 2024-03-29T11:26:26.435 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:26:26.435 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:26:26.435 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:26.435 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:26.435 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:26.435 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:26:26.435 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:26:26.436 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:26:26.436 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:26:26.436 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:26:26.436 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.446+0000 7f6ddffff640 1 -- 172.21.15.104:0/3452494388 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f6dec003b60 con 0x7f6df8141fb0 2024-03-29T11:26:26.436 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.446+0000 7f6dfe231640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:26.436 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.446+0000 7f6dfe231640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:26.436 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.446+0000 7f6dfe231640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:26:26.436 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.446+0000 7f6dfe231640 1 -- 172.21.15.104:0/3452494388 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6df8075f10 con 0x7f6df8141fb0 2024-03-29T11:26:26.436 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.446+0000 7f6df5ffb640 4 set_mon_vals no callback set 2024-03-29T11:26:26.436 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.447+0000 7f6ddffff640 1 -- 172.21.15.104:0/3452494388 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f6dec084960 con 0x7f6df8141fb0 2024-03-29T11:26:26.436 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.447+0000 7f6ddffff640 10 client.4471.objecter ms_dispatch 0x7f6df8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:26.436 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.447+0000 7f6ddffff640 3 client.4471.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:26.436 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.447+0000 7f6ddffff640 3 client.4471.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:26.437 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.447+0000 7f6ddffff640 20 client.4471.objecter dump_active .. 0 homeless 2024-03-29T11:26:26.437 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.447+0000 7f6df77fe640 10 monclient: get_auth_request con 0x7f6de406de00 auth_method 0 2024-03-29T11:26:26.437 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.447+0000 7f6ddffff640 10 client.4471.objecter ms_handle_connect 0x7f6de406de00 2024-03-29T11:26:26.441 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.455+0000 7f6ddffff640 1 -- 172.21.15.104:0/3452494388 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6dec050240 con 0x7f6df8141fb0 2024-03-29T11:26:26.442 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.456+0000 7f6ddffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:26.442 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.456+0000 7f6ddffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:26.768 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.780+0000 7f6dfe231640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:26.768 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.780+0000 7f6dfe231640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:26.768 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.780+0000 7f6dfe231640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:26:26.769 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.780+0000 7f6dfe231640 1 -- 172.21.15.104:0/3452494388 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6df80b2e60 con 0x7f6df8141fb0 2024-03-29T11:26:26.769 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.782+0000 7f6ddffff640 1 -- 172.21.15.104:0/3452494388 <== mon.1 v1:172.21.15.119:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f6dec052170 con 0x7f6df8141fb0 2024-03-29T11:26:26.769 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.782+0000 7f6ddffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:26.769 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.782+0000 7f6ddffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:26.769 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:26:26.769 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:26.772 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.786+0000 7f6dfe231640 10 librados: watch_flush enter 2024-03-29T11:26:26.772 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.786+0000 7f6dfe231640 10 librados: watch_flush exit 2024-03-29T11:26:26.773 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.786+0000 7f6dfe231640 20 client.4471.objecter shutdown clearing up homeless session... 2024-03-29T11:26:26.773 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.786+0000 7f6dfe231640 10 client.4471.objecter successfully canceled tick 2024-03-29T11:26:26.773 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.787+0000 7f6dfe231640 1 -- 172.21.15.104:0/3452494388 >> v1:172.21.15.104:6816/3341126660 conn(0x7f6de406de00 legacy=0x7f6de40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:26.773 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.787+0000 7f6dfe231640 10 monclient: shutdown 2024-03-29T11:26:26.773 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.787+0000 7f6dfe231640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:26.773 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.787+0000 7f6dfe231640 1 -- 172.21.15.104:0/3452494388 >> v1:172.21.15.119:6789/0 conn(0x7f6df8141fb0 legacy=0x7f6df81e79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:26.773 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.787+0000 7f6dfe231640 1 -- 172.21.15.104:0/3452494388 shutdown_connections 2024-03-29T11:26:26.773 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.787+0000 7f6dfe231640 1 -- 172.21.15.104:0/3452494388 >> 172.21.15.104:0/3452494388 conn(0x7f6df8067470 msgr2=0x7f6df8069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:26.774 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.787+0000 7f6dfe231640 1 -- 172.21.15.104:0/3452494388 shutdown_connections 2024-03-29T11:26:26.774 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.787+0000 7f6dfe231640 1 -- 172.21.15.104:0/3452494388 wait complete. 2024-03-29T11:26:26.774 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:26.787+0000 7f6dfe231640 1 librados: shutdown 2024-03-29T11:26:26.791 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:33.598 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:33.752 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.766+0000 7f0d067f4640 10 monclient: get_monmap_and_config 2024-03-29T11:26:33.752 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.766+0000 7f0d067f4640 10 monclient: build_initial_monmap 2024-03-29T11:26:33.752 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.766+0000 7f0d067f4640 10 monclient: monmap: 2024-03-29T11:26:33.752 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:26:33.753 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:26:33.753 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:26:33.767187+0000 2024-03-29T11:26:33.753 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:26:33.767187+0000 2024-03-29T11:26:33.753 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:33.753 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:26:33.753 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:26:33.753 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:26:33.753 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:26:33.753 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:26:33.754 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.767+0000 7f0d067f4640 1 Processor -- start 2024-03-29T11:26:33.754 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.767+0000 7f0d067f4640 1 -- start start 2024-03-29T11:26:33.754 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.767+0000 7f0d067f4640 10 monclient: init 2024-03-29T11:26:33.756 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.768+0000 7f0d067f4640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:33.756 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.768+0000 7f0d067f4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:26:33.756 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.768+0000 7f0d067f4640 10 monclient(hunting): picked mon.noname-c con 0x7f0d001457f0 addr v1:172.21.15.119:6789/0 2024-03-29T11:26:33.756 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.768+0000 7f0d067f4640 10 monclient(hunting): picked mon.noname-b con 0x7f0d0013f650 addr v1:172.21.15.104:6790/0 2024-03-29T11:26:33.756 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.768+0000 7f0d067f4640 10 monclient(hunting): picked mon.noname-a con 0x7f0d000a9d70 addr v1:172.21.15.104:6789/0 2024-03-29T11:26:33.756 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.768+0000 7f0d067f4640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d000aaef0 con 0x7f0d000a9d70 2024-03-29T11:26:33.757 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.768+0000 7f0d067f4640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d00145bd0 con 0x7f0d001457f0 2024-03-29T11:26:33.757 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.768+0000 7f0d067f4640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d00145eb0 con 0x7f0d0013f650 2024-03-29T11:26:33.757 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.768+0000 7f0d067f4640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:33.757 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.768+0000 7f0d067f4640 10 monclient(hunting): authenticate will time out at 1375.019783s 2024-03-29T11:26:33.760 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.772+0000 7f0cff7fe640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f0d0013f650 0x7f0d000b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:43583/0 (socket says 172.21.15.104:43583) 2024-03-29T11:26:33.760 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.772+0000 7f0cff7fe640 1 -- 172.21.15.104:0/520624430 learned_addr learned my addr 172.21.15.104:0/520624430 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:26:33.760 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.774+0000 7f0cfeffd640 1 -- 172.21.15.104:0/520624430 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1422813377 0 0) 0x7f0d00145eb0 con 0x7f0d0013f650 2024-03-29T11:26:33.760 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.774+0000 7f0cfeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:33.761 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.774+0000 7f0cfeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:33.761 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.774+0000 7f0cfeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:33.761 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.774+0000 7f0cfeffd640 10 monclient(hunting): my global_id is 4502 2024-03-29T11:26:33.761 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.774+0000 7f0cfeffd640 1 -- 172.21.15.104:0/520624430 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ce8003520 con 0x7f0d0013f650 2024-03-29T11:26:33.763 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.775+0000 7f0cfeffd640 1 -- 172.21.15.104:0/520624430 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 369632398 0 0) 0x7f0ce8003520 con 0x7f0d0013f650 2024-03-29T11:26:33.763 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.775+0000 7f0cfeffd640 1 -- 172.21.15.104:0/520624430 >> v1:172.21.15.119:6789/0 conn(0x7f0d001457f0 legacy=0x7f0d0013ef40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:26:33.763 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.775+0000 7f0cfeffd640 1 -- 172.21.15.104:0/520624430 >> v1:172.21.15.104:6789/0 conn(0x7f0d000a9d70 legacy=0x7f0d0014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:33.763 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.775+0000 7f0cfeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:26:33.763 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.775+0000 7f0cfeffd640 1 monclient: found mon.noname-b 2024-03-29T11:26:33.763 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.775+0000 7f0cfeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:33.763 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.775+0000 7f0cfeffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.104:6790/0 2024-03-29T11:26:33.763 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.775+0000 7f0cfeffd640 1 -- 172.21.15.104:0/520624430 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d00146340 con 0x7f0d0013f650 2024-03-29T11:26:33.763 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.775+0000 7f0cfeffd640 10 monclient: _finish_auth 0 2024-03-29T11:26:33.764 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.775+0000 7f0cfeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:26:33.764 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.775+0000 7f0cfeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:33.764 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.775+0000 7f0cfeffd640 1 -- 172.21.15.104:0/520624430 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f0cf0001690 con 0x7f0d0013f650 2024-03-29T11:26:33.764 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.775+0000 7f0cfeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:33.764 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.775+0000 7f0cfeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:33.764 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.775+0000 7f0cfeffd640 10 monclient: dump: 2024-03-29T11:26:33.764 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:26:33.764 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:26:33.764 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:33.764 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:33.764 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:33.764 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:26:33.764 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:26:33.764 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:26:33.764 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:26:33.765 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:26:33.765 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.776+0000 7f0d067f4640 5 monclient: authenticate success, global_id 4502 2024-03-29T11:26:33.765 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.776+0000 7f0d067f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:33.765 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.777+0000 7f0cfeffd640 1 -- 172.21.15.104:0/520624430 <== mon.2 v1:172.21.15.104:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0cf0001860 con 0x7f0d0013f650 2024-03-29T11:26:33.765 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.777+0000 7f0cfeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:33.765 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.777+0000 7f0d067f4640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:33.765 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.777+0000 7f0d067f4640 4 set_mon_vals no callback set 2024-03-29T11:26:33.765 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.778+0000 7f0d067f4640 10 monclient: shutdown 2024-03-29T11:26:33.766 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.778+0000 7f0d067f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:33.766 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.778+0000 7f0d067f4640 1 -- 172.21.15.104:0/520624430 >> v1:172.21.15.104:6790/0 conn(0x7f0d0013f650 legacy=0x7f0d000b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:33.766 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.778+0000 7f0d067f4640 1 -- 172.21.15.104:0/520624430 shutdown_connections 2024-03-29T11:26:33.766 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.779+0000 7f0d067f4640 1 -- 172.21.15.104:0/520624430 >> 172.21.15.104:0/520624430 conn(0x7f0d00066a00 msgr2=0x7f0d00066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:33.766 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.779+0000 7f0d067f4640 1 -- 172.21.15.104:0/520624430 shutdown_connections 2024-03-29T11:26:33.766 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.779+0000 7f0d067f4640 1 -- 172.21.15.104:0/520624430 wait complete. 2024-03-29T11:26:33.766 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.780+0000 7f0d067f4640 10 monclient: build_initial_monmap 2024-03-29T11:26:33.766 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.780+0000 7f0d067f4640 10 monclient: monmap: 2024-03-29T11:26:33.766 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:26:33.766 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:33.767 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:26:33.767 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:26:33.767 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:33.767 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:26:33.767 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:26:33.767 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:26:33.767 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:26:33.767 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:26:33.768 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.780+0000 7f0d067f4640 1 librados: starting msgr at 2024-03-29T11:26:33.768 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.780+0000 7f0d067f4640 1 librados: starting objecter 2024-03-29T11:26:33.769 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.781+0000 7f0d067f4640 1 Processor -- start 2024-03-29T11:26:33.769 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.781+0000 7f0d067f4640 1 -- start start 2024-03-29T11:26:33.769 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.781+0000 7f0d067f4640 1 librados: setting wanted keys 2024-03-29T11:26:33.769 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.781+0000 7f0d067f4640 1 librados: calling monclient init 2024-03-29T11:26:33.769 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.781+0000 7f0d067f4640 10 monclient: init 2024-03-29T11:26:33.769 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.782+0000 7f0d067f4640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:33.769 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.782+0000 7f0d067f4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:26:33.769 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.782+0000 7f0d067f4640 10 monclient(hunting): picked mon.noname-b con 0x7f0d000a9d70 addr v1:172.21.15.119:6789/0 2024-03-29T11:26:33.769 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.782+0000 7f0d067f4640 10 monclient(hunting): picked mon.noname-a con 0x7f0d0013f650 addr v1:172.21.15.104:6789/0 2024-03-29T11:26:33.769 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.782+0000 7f0d067f4640 10 monclient(hunting): picked mon.noname-c con 0x7f0d001457f0 addr v1:172.21.15.104:6790/0 2024-03-29T11:26:33.769 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.782+0000 7f0d067f4640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d001389a0 con 0x7f0d0013f650 2024-03-29T11:26:33.769 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.782+0000 7f0d067f4640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d001db330 con 0x7f0d000a9d70 2024-03-29T11:26:33.769 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.782+0000 7f0d067f4640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d001da700 con 0x7f0d001457f0 2024-03-29T11:26:33.769 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.782+0000 7f0d067f4640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:33.769 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.782+0000 7f0d067f4640 10 monclient(hunting): authenticate will time out at 1375.033173s 2024-03-29T11:26:33.772 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.786+0000 7f0cff7fe640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f0d0013f650 0x7f0d00137b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:51749/0 (socket says 172.21.15.104:51749) 2024-03-29T11:26:33.772 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.786+0000 7f0cff7fe640 1 -- 172.21.15.104:0/3092259781 learned_addr learned my addr 172.21.15.104:0/3092259781 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:26:33.772 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.786+0000 7f0cfcff9640 10 client.?.objecter ms_handle_connect 0x7f0d0013f650 2024-03-29T11:26:33.772 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.786+0000 7f0cfcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:33.773 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.786+0000 7f0cfcff9640 1 -- 172.21.15.104:0/3092259781 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3986258446 0 0) 0x7f0d001389a0 con 0x7f0d0013f650 2024-03-29T11:26:33.773 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.786+0000 7f0cfcff9640 10 monclient(hunting): _negotiate 2024-03-29T11:26:33.773 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.786+0000 7f0cfcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:33.773 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.786+0000 7f0cfcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:33.773 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.786+0000 7f0cfcff9640 10 monclient(hunting): my global_id is 4497 2024-03-29T11:26:33.773 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.786+0000 7f0cfcff9640 1 -- 172.21.15.104:0/3092259781 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cec004eb0 con 0x7f0d0013f650 2024-03-29T11:26:33.775 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.789+0000 7f0cfcff9640 10 client.?.objecter ms_handle_connect 0x7f0d000a9d70 2024-03-29T11:26:33.775 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.789+0000 7f0cfcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:33.776 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.790+0000 7f0cfcff9640 10 client.?.objecter ms_handle_connect 0x7f0d001457f0 2024-03-29T11:26:33.776 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.790+0000 7f0cfcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:33.776 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.790+0000 7f0cfcff9640 1 -- 172.21.15.104:0/3092259781 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2957308588 0 0) 0x7f0d001da700 con 0x7f0d001457f0 2024-03-29T11:26:33.776 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.790+0000 7f0cfcff9640 10 monclient(hunting): _negotiate 2024-03-29T11:26:33.776 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.790+0000 7f0cfcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:33.777 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.790+0000 7f0cfcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:33.777 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.790+0000 7f0cfcff9640 10 monclient(hunting): my global_id is 4505 2024-03-29T11:26:33.777 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.790+0000 7f0cfcff9640 1 -- 172.21.15.104:0/3092259781 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cec0051d0 con 0x7f0d001457f0 2024-03-29T11:26:33.777 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.790+0000 7f0cfcff9640 1 -- 172.21.15.104:0/3092259781 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 312714571 0 0) 0x7f0cec004eb0 con 0x7f0d0013f650 2024-03-29T11:26:33.777 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.790+0000 7f0cfcff9640 1 -- 172.21.15.104:0/3092259781 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0cec005640 con 0x7f0d0013f650 2024-03-29T11:26:33.777 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.790+0000 7f0cfcff9640 1 -- 172.21.15.104:0/3092259781 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f0cf0004ae0 con 0x7f0d0013f650 2024-03-29T11:26:33.777 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.790+0000 7f0cfcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:33.777 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.790+0000 7f0cfcff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:33.777 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.790+0000 7f0cfcff9640 10 monclient(hunting): dump: 2024-03-29T11:26:33.777 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:26:33.777 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:26:33.777 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:33.777 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:33.777 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:33.778 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:26:33.778 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:26:33.778 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:26:33.778 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:26:33.778 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:26:33.778 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.790+0000 7f0cfcff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:33.779 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.791+0000 7f0cfcff9640 1 -- 172.21.15.104:0/3092259781 <== mon.0 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1515602551 0 0) 0x7f0cec005640 con 0x7f0d0013f650 2024-03-29T11:26:33.779 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.791+0000 7f0cfcff9640 1 -- 172.21.15.104:0/3092259781 >> v1:172.21.15.104:6790/0 conn(0x7f0d001457f0 legacy=0x7f0d00138290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:33.779 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.791+0000 7f0cfcff9640 1 -- 172.21.15.104:0/3092259781 >> v1:172.21.15.119:6789/0 conn(0x7f0d000a9d70 legacy=0x7f0d001dac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:33.779 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.791+0000 7f0cfcff9640 10 monclient: _finish_hunting 0 2024-03-29T11:26:33.779 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.791+0000 7f0cfcff9640 1 monclient: found mon.a 2024-03-29T11:26:33.779 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.791+0000 7f0cfcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:33.779 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.791+0000 7f0cfcff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:26:33.779 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.791+0000 7f0cfcff9640 1 -- 172.21.15.104:0/3092259781 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d00142c70 con 0x7f0d0013f650 2024-03-29T11:26:33.779 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.791+0000 7f0cfcff9640 10 monclient: _finish_auth 0 2024-03-29T11:26:33.780 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.791+0000 7f0cfcff9640 10 monclient: _check_auth_tickets 2024-03-29T11:26:33.780 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.791+0000 7f0cfcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:33.780 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.791+0000 7f0d067f4640 5 monclient: authenticate success, global_id 4497 2024-03-29T11:26:33.780 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.791+0000 7f0d067f4640 10 monclient: _renew_subs 2024-03-29T11:26:33.780 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.791+0000 7f0d067f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:26:33.780 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.791+0000 7f0d067f4640 1 -- 172.21.15.104:0/3092259781 --> v1:172.21.15.104:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d00142610 con 0x7f0d0013f650 2024-03-29T11:26:33.780 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.791+0000 7f0d067f4640 10 client.4497.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:33.780 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.791+0000 7f0d067f4640 10 monclient: _renew_subs 2024-03-29T11:26:33.780 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.791+0000 7f0d067f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:26:33.780 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.791+0000 7f0d067f4640 1 -- 172.21.15.104:0/3092259781 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d0013ff80 con 0x7f0d0013f650 2024-03-29T11:26:33.780 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.791+0000 7f0d067f4640 1 librados: init done 2024-03-29T11:26:33.780 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.792+0000 7f0d067f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:33.781 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.795+0000 7f0d067f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:33.781 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.795+0000 7f0d067f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:26:33.782 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.795+0000 7f0d067f4640 1 -- 172.21.15.104:0/3092259781 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d001429b0 con 0x7f0d0013f650 2024-03-29T11:26:33.782 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.796+0000 7f0cfcff9640 1 -- 172.21.15.104:0/3092259781 <== mon.0 v1:172.21.15.104:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0cf0001b10 con 0x7f0d0013f650 2024-03-29T11:26:33.783 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.796+0000 7f0cfcff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:33.783 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.796+0000 7f0cfcff9640 1 -- 172.21.15.104:0/3092259781 <== mon.0 v1:172.21.15.104:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f0cf0004ae0 con 0x7f0d0013f650 2024-03-29T11:26:33.783 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.796+0000 7f0cfcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:33.783 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.796+0000 7f0cfcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:33.783 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.796+0000 7f0cfcff9640 10 monclient: dump: 2024-03-29T11:26:33.783 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:26:33.783 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:26:33.783 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:33.783 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:33.783 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:33.783 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:26:33.783 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:26:33.783 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:26:33.783 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:26:33.784 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:26:33.784 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.796+0000 7f0cfcff9640 1 -- 172.21.15.104:0/3092259781 <== mon.0 v1:172.21.15.104:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f0cf001ad60 con 0x7f0d0013f650 2024-03-29T11:26:33.784 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.797+0000 7f0cfffff640 10 monclient: get_auth_request con 0x7f0cec06e330 auth_method 0 2024-03-29T11:26:33.784 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.796+0000 7f0cfdffb640 4 set_mon_vals no callback set 2024-03-29T11:26:33.784 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.798+0000 7f0cfcff9640 10 client.4497.objecter ms_handle_connect 0x7f0cec06e330 2024-03-29T11:26:33.787 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.799+0000 7f0cfcff9640 1 -- 172.21.15.104:0/3092259781 <== mon.0 v1:172.21.15.104:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f0cf0084990 con 0x7f0d0013f650 2024-03-29T11:26:33.787 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.800+0000 7f0cfcff9640 10 client.4497.objecter ms_dispatch 0x7f0d00000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:33.787 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.800+0000 7f0cfcff9640 3 client.4497.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:33.787 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.800+0000 7f0cfcff9640 3 client.4497.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:33.787 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.800+0000 7f0cfcff9640 20 client.4497.objecter dump_active .. 0 homeless 2024-03-29T11:26:33.797 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.806+0000 7f0cfcff9640 1 -- 172.21.15.104:0/3092259781 <== mon.0 v1:172.21.15.104:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0cf00503b0 con 0x7f0d0013f650 2024-03-29T11:26:33.797 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.806+0000 7f0cfcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:33.798 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:33.806+0000 7f0cfcff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:34.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:34.126+0000 7f0d067f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:34.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:34.126+0000 7f0d067f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:34.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:34.126+0000 7f0d067f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:26:34.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:34.126+0000 7f0d067f4640 1 -- 172.21.15.104:0/3092259781 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d001429b0 con 0x7f0d0013f650 2024-03-29T11:26:34.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:34.127+0000 7f0cfcff9640 1 -- 172.21.15.104:0/3092259781 <== mon.0 v1:172.21.15.104:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f0cf00522e0 con 0x7f0d0013f650 2024-03-29T11:26:34.114 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:34.127+0000 7f0cfcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:34.114 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:34.127+0000 7f0cfcff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:34.115 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:26:34.115 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:34.119 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:34.132+0000 7f0d067f4640 10 librados: watch_flush enter 2024-03-29T11:26:34.119 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:34.132+0000 7f0d067f4640 10 librados: watch_flush exit 2024-03-29T11:26:34.119 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:34.132+0000 7f0d067f4640 20 client.4497.objecter shutdown clearing up homeless session... 2024-03-29T11:26:34.119 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:34.132+0000 7f0d067f4640 10 client.4497.objecter successfully canceled tick 2024-03-29T11:26:34.119 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:34.132+0000 7f0d067f4640 1 -- 172.21.15.104:0/3092259781 >> v1:172.21.15.104:6816/3341126660 conn(0x7f0cec06e330 legacy=0x7f0cec0707f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:34.119 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:34.132+0000 7f0d067f4640 10 monclient: shutdown 2024-03-29T11:26:34.119 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:34.132+0000 7f0d067f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:34.120 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:34.132+0000 7f0d067f4640 1 -- 172.21.15.104:0/3092259781 >> v1:172.21.15.104:6789/0 conn(0x7f0d0013f650 legacy=0x7f0d00137b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:34.120 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:34.132+0000 7f0d067f4640 1 -- 172.21.15.104:0/3092259781 shutdown_connections 2024-03-29T11:26:34.120 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:34.132+0000 7f0d067f4640 1 -- 172.21.15.104:0/3092259781 >> 172.21.15.104:0/3092259781 conn(0x7f0d00066a00 msgr2=0x7f0d0013c2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:34.122 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:34.135+0000 7f0d067f4640 1 -- 172.21.15.104:0/3092259781 shutdown_connections 2024-03-29T11:26:34.122 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:34.135+0000 7f0d067f4640 1 -- 172.21.15.104:0/3092259781 wait complete. 2024-03-29T11:26:34.122 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:34.135+0000 7f0d067f4640 1 librados: shutdown 2024-03-29T11:26:34.139 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:40.947 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:41.111 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.124+0000 7f1bbb324640 10 monclient: get_monmap_and_config 2024-03-29T11:26:41.111 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.124+0000 7f1bbb324640 10 monclient: build_initial_monmap 2024-03-29T11:26:41.111 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.124+0000 7f1bbb324640 10 monclient: monmap: 2024-03-29T11:26:41.111 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:26:41.112 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:26:41.112 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:26:41.125871+0000 2024-03-29T11:26:41.112 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:26:41.125871+0000 2024-03-29T11:26:41.112 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:41.112 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:26:41.112 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:26:41.112 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:26:41.112 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:26:41.112 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:26:41.114 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.126+0000 7f1bbb324640 1 Processor -- start 2024-03-29T11:26:41.114 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.126+0000 7f1bbb324640 1 -- start start 2024-03-29T11:26:41.114 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.126+0000 7f1bbb324640 10 monclient: init 2024-03-29T11:26:41.114 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.127+0000 7f1bbb324640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:41.114 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.127+0000 7f1bbb324640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:26:41.114 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.127+0000 7f1bbb324640 10 monclient(hunting): picked mon.noname-c con 0x7f1bb41457f0 addr v1:172.21.15.119:6789/0 2024-03-29T11:26:41.114 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.127+0000 7f1bbb324640 10 monclient(hunting): picked mon.noname-b con 0x7f1bb413f650 addr v1:172.21.15.104:6790/0 2024-03-29T11:26:41.115 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.127+0000 7f1bbb324640 10 monclient(hunting): picked mon.noname-a con 0x7f1bb40a9d70 addr v1:172.21.15.104:6789/0 2024-03-29T11:26:41.115 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.127+0000 7f1bbb324640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bb40aaef0 con 0x7f1bb40a9d70 2024-03-29T11:26:41.115 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.127+0000 7f1bbb324640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bb4145bd0 con 0x7f1bb41457f0 2024-03-29T11:26:41.115 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.127+0000 7f1bbb324640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bb4145eb0 con 0x7f1bb413f650 2024-03-29T11:26:41.115 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.127+0000 7f1bbb324640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:41.115 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.127+0000 7f1bbb324640 10 monclient(hunting): authenticate will time out at 1382.378540s 2024-03-29T11:26:41.121 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.135+0000 7f1bb88bf640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f1bb413f650 0x7f1bb40b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:40739/0 (socket says 172.21.15.104:40739) 2024-03-29T11:26:41.121 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.135+0000 7f1bb88bf640 1 -- 172.21.15.104:0/712840287 learned_addr learned my addr 172.21.15.104:0/712840287 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:26:41.124 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.136+0000 7f1babfff640 1 -- 172.21.15.104:0/712840287 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 863173276 0 0) 0x7f1bb4145bd0 con 0x7f1bb41457f0 2024-03-29T11:26:41.124 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.136+0000 7f1babfff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:41.124 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.136+0000 7f1babfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:41.124 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.136+0000 7f1babfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:41.124 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.136+0000 7f1babfff640 10 monclient(hunting): my global_id is 4477 2024-03-29T11:26:41.124 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.136+0000 7f1babfff640 1 -- 172.21.15.104:0/712840287 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ba0003520 con 0x7f1bb41457f0 2024-03-29T11:26:41.125 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.138+0000 7f1babfff640 1 -- 172.21.15.104:0/712840287 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1009240125 0 0) 0x7f1bb4145eb0 con 0x7f1bb413f650 2024-03-29T11:26:41.125 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.138+0000 7f1babfff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:41.125 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.138+0000 7f1babfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:41.125 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.138+0000 7f1babfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:41.125 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.138+0000 7f1babfff640 10 monclient(hunting): my global_id is 4508 2024-03-29T11:26:41.125 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.138+0000 7f1babfff640 1 -- 172.21.15.104:0/712840287 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bb4145bd0 con 0x7f1bb413f650 2024-03-29T11:26:41.125 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.138+0000 7f1babfff640 1 -- 172.21.15.104:0/712840287 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2285754249 0 0) 0x7f1ba0003520 con 0x7f1bb41457f0 2024-03-29T11:26:41.125 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.138+0000 7f1babfff640 1 -- 172.21.15.104:0/712840287 >> v1:172.21.15.104:6790/0 conn(0x7f1bb413f650 legacy=0x7f1bb40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:41.125 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.138+0000 7f1babfff640 1 -- 172.21.15.104:0/712840287 >> v1:172.21.15.104:6789/0 conn(0x7f1bb40a9d70 legacy=0x7f1bb414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:41.125 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.138+0000 7f1babfff640 10 monclient: _finish_hunting 0 2024-03-29T11:26:41.125 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.138+0000 7f1babfff640 1 monclient: found mon.noname-c 2024-03-29T11:26:41.125 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.138+0000 7f1babfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:41.125 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.138+0000 7f1babfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.119:6789/0 2024-03-29T11:26:41.125 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.138+0000 7f1babfff640 1 -- 172.21.15.104:0/712840287 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bb4146340 con 0x7f1bb41457f0 2024-03-29T11:26:41.126 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.138+0000 7f1babfff640 10 monclient: _finish_auth 0 2024-03-29T11:26:41.126 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.138+0000 7f1babfff640 10 monclient: _check_auth_tickets 2024-03-29T11:26:41.126 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.138+0000 7f1babfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:41.126 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.138+0000 7f1babfff640 1 -- 172.21.15.104:0/712840287 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f1bb0001690 con 0x7f1bb41457f0 2024-03-29T11:26:41.126 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.138+0000 7f1babfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:41.126 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.138+0000 7f1babfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:41.126 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.138+0000 7f1babfff640 10 monclient: dump: 2024-03-29T11:26:41.126 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:26:41.126 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:26:41.126 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:41.126 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:41.126 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:41.126 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:26:41.126 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:26:41.127 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:26:41.127 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:26:41.127 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:26:41.128 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.138+0000 7f1babfff640 1 -- 172.21.15.104:0/712840287 <== mon.1 v1:172.21.15.119:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1bb00017f0 con 0x7f1bb41457f0 2024-03-29T11:26:41.128 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.138+0000 7f1babfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:41.128 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.138+0000 7f1babfff640 1 -- 172.21.15.104:0/712840287 <== mon.1 v1:172.21.15.119:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f1bb0001c00 con 0x7f1bb41457f0 2024-03-29T11:26:41.128 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.138+0000 7f1babfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:41.128 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.138+0000 7f1babfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:41.128 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.138+0000 7f1babfff640 10 monclient: dump: 2024-03-29T11:26:41.128 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:26:41.128 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:26:41.128 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:41.128 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:41.128 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:41.128 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:26:41.128 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:26:41.129 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:26:41.129 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:26:41.129 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:26:41.129 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.138+0000 7f1bbb324640 5 monclient: authenticate success, global_id 4477 2024-03-29T11:26:41.129 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.138+0000 7f1bbb324640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:41.129 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.138+0000 7f1bbb324640 4 set_mon_vals no callback set 2024-03-29T11:26:41.129 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.139+0000 7f1bbb324640 10 monclient: shutdown 2024-03-29T11:26:41.129 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.139+0000 7f1bbb324640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:41.129 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.139+0000 7f1bbb324640 1 -- 172.21.15.104:0/712840287 >> v1:172.21.15.119:6789/0 conn(0x7f1bb41457f0 legacy=0x7f1bb413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:41.129 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.141+0000 7f1bbb324640 1 -- 172.21.15.104:0/712840287 shutdown_connections 2024-03-29T11:26:41.129 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.141+0000 7f1bbb324640 1 -- 172.21.15.104:0/712840287 >> 172.21.15.104:0/712840287 conn(0x7f1bb4066a00 msgr2=0x7f1bb4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:41.131 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.144+0000 7f1bbb324640 1 -- 172.21.15.104:0/712840287 shutdown_connections 2024-03-29T11:26:41.131 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.145+0000 7f1bbb324640 1 -- 172.21.15.104:0/712840287 wait complete. 2024-03-29T11:26:41.133 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.146+0000 7f1bbb324640 10 monclient: build_initial_monmap 2024-03-29T11:26:41.136 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.146+0000 7f1bbb324640 10 monclient: monmap: 2024-03-29T11:26:41.136 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:26:41.136 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:41.137 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:26:41.137 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:26:41.137 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:41.137 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:26:41.137 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:26:41.137 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:26:41.137 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:26:41.137 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:26:41.137 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.146+0000 7f1bbb324640 1 librados: starting msgr at 2024-03-29T11:26:41.137 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.146+0000 7f1bbb324640 1 librados: starting objecter 2024-03-29T11:26:41.137 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.146+0000 7f1bbb324640 1 Processor -- start 2024-03-29T11:26:41.137 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.146+0000 7f1bbb324640 1 -- start start 2024-03-29T11:26:41.137 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.146+0000 7f1bbb324640 1 librados: setting wanted keys 2024-03-29T11:26:41.137 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.146+0000 7f1bbb324640 1 librados: calling monclient init 2024-03-29T11:26:41.137 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.146+0000 7f1bbb324640 10 monclient: init 2024-03-29T11:26:41.138 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.147+0000 7f1bbb324640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:41.138 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.147+0000 7f1bbb324640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:26:41.138 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.147+0000 7f1bbb324640 10 monclient(hunting): picked mon.noname-b con 0x7f1bb40a9d70 addr v1:172.21.15.119:6789/0 2024-03-29T11:26:41.138 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.147+0000 7f1bbb324640 10 monclient(hunting): picked mon.noname-c con 0x7f1bb413f650 addr v1:172.21.15.104:6790/0 2024-03-29T11:26:41.138 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.147+0000 7f1bbb324640 10 monclient(hunting): picked mon.noname-a con 0x7f1bb41457f0 addr v1:172.21.15.104:6789/0 2024-03-29T11:26:41.138 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.147+0000 7f1bbb324640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bb4142d30 con 0x7f1bb41457f0 2024-03-29T11:26:41.138 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.147+0000 7f1bbb324640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bb406e810 con 0x7f1bb40a9d70 2024-03-29T11:26:41.138 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.147+0000 7f1bbb324640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bb41429e0 con 0x7f1bb413f650 2024-03-29T11:26:41.138 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.147+0000 7f1bbb324640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:41.138 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.147+0000 7f1bbb324640 10 monclient(hunting): authenticate will time out at 1382.398716s 2024-03-29T11:26:41.146 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.151+0000 7f1bb98c1640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f1bb41457f0 0x7f1bb4140790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:43167/0 (socket says 172.21.15.104:43167) 2024-03-29T11:26:41.146 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.151+0000 7f1bb98c1640 1 -- 172.21.15.104:0/3098813976 learned_addr learned my addr 172.21.15.104:0/3098813976 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:26:41.146 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.151+0000 7f1ba9ffb640 10 client.?.objecter ms_handle_connect 0x7f1bb41457f0 2024-03-29T11:26:41.146 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.151+0000 7f1ba9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:41.146 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.152+0000 7f1ba9ffb640 10 client.?.objecter ms_handle_connect 0x7f1bb40a9d70 2024-03-29T11:26:41.146 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.152+0000 7f1ba9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:41.146 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.152+0000 7f1ba9ffb640 1 -- 172.21.15.104:0/3098813976 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3106609806 0 0) 0x7f1bb4142d30 con 0x7f1bb41457f0 2024-03-29T11:26:41.146 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.152+0000 7f1ba9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:41.146 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.152+0000 7f1ba9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:41.146 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.152+0000 7f1ba9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:41.146 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.152+0000 7f1ba9ffb640 10 monclient(hunting): my global_id is 4503 2024-03-29T11:26:41.147 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.152+0000 7f1ba9ffb640 1 -- 172.21.15.104:0/3098813976 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b98004eb0 con 0x7f1bb41457f0 2024-03-29T11:26:41.147 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.153+0000 7f1ba9ffb640 10 client.?.objecter ms_handle_connect 0x7f1bb413f650 2024-03-29T11:26:41.147 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.153+0000 7f1ba9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:41.147 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.153+0000 7f1ba9ffb640 1 -- 172.21.15.104:0/3098813976 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4200328332 0 0) 0x7f1b98004eb0 con 0x7f1bb41457f0 2024-03-29T11:26:41.147 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.153+0000 7f1ba9ffb640 1 -- 172.21.15.104:0/3098813976 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1bb4142d30 con 0x7f1bb41457f0 2024-03-29T11:26:41.147 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.153+0000 7f1ba9ffb640 1 -- 172.21.15.104:0/3098813976 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f1bac001cd0 con 0x7f1bb41457f0 2024-03-29T11:26:41.147 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.153+0000 7f1ba9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:41.147 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.153+0000 7f1ba9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:41.147 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.153+0000 7f1ba9ffb640 10 monclient(hunting): dump: 2024-03-29T11:26:41.147 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:26:41.147 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:26:41.147 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:41.147 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:41.147 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:41.148 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:26:41.148 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:26:41.148 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:26:41.148 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:26:41.148 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:26:41.148 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.153+0000 7f1ba9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:41.148 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.153+0000 7f1ba9ffb640 1 -- 172.21.15.104:0/3098813976 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4230617507 0 0) 0x7f1bb41429e0 con 0x7f1bb413f650 2024-03-29T11:26:41.148 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.153+0000 7f1ba9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:41.148 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.153+0000 7f1ba9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:41.148 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.153+0000 7f1ba9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:41.148 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.153+0000 7f1ba9ffb640 10 monclient(hunting): my global_id is 4511 2024-03-29T11:26:41.148 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.153+0000 7f1ba9ffb640 1 -- 172.21.15.104:0/3098813976 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b98004eb0 con 0x7f1bb413f650 2024-03-29T11:26:41.148 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.153+0000 7f1ba9ffb640 1 -- 172.21.15.104:0/3098813976 <== mon.0 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 928255143 0 0) 0x7f1bb4142d30 con 0x7f1bb41457f0 2024-03-29T11:26:41.148 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.153+0000 7f1ba9ffb640 1 -- 172.21.15.104:0/3098813976 >> v1:172.21.15.104:6790/0 conn(0x7f1bb413f650 legacy=0x7f1bb4140080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:41.148 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.153+0000 7f1ba9ffb640 1 -- 172.21.15.104:0/3098813976 >> v1:172.21.15.119:6789/0 conn(0x7f1bb40a9d70 legacy=0x7f1bb406e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:41.149 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.153+0000 7f1ba9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:26:41.149 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.153+0000 7f1ba9ffb640 1 monclient: found mon.a 2024-03-29T11:26:41.149 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.153+0000 7f1ba9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:41.149 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.153+0000 7f1ba9ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:26:41.149 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.153+0000 7f1ba9ffb640 1 -- 172.21.15.104:0/3098813976 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bb406f540 con 0x7f1bb41457f0 2024-03-29T11:26:41.149 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.153+0000 7f1ba9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:26:41.149 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.153+0000 7f1ba9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:26:41.149 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.153+0000 7f1ba9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:41.149 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.153+0000 7f1bbb324640 5 monclient: authenticate success, global_id 4503 2024-03-29T11:26:41.149 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.153+0000 7f1bbb324640 10 monclient: _renew_subs 2024-03-29T11:26:41.149 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.153+0000 7f1bbb324640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:26:41.149 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.153+0000 7f1bbb324640 1 -- 172.21.15.104:0/3098813976 --> v1:172.21.15.104:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bb406f820 con 0x7f1bb41457f0 2024-03-29T11:26:41.149 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.154+0000 7f1bbb324640 10 client.4503.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:41.149 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.154+0000 7f1bbb324640 10 monclient: _renew_subs 2024-03-29T11:26:41.149 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.154+0000 7f1bbb324640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:26:41.150 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.154+0000 7f1bbb324640 1 -- 172.21.15.104:0/3098813976 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bb41efc40 con 0x7f1bb41457f0 2024-03-29T11:26:41.150 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.154+0000 7f1bbb324640 1 librados: init done 2024-03-29T11:26:41.150 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.155+0000 7f1bbb324640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:41.150 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.155+0000 7f1bbb324640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:41.150 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.155+0000 7f1bbb324640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:26:41.150 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.155+0000 7f1bbb324640 1 -- 172.21.15.104:0/3098813976 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bb4075f10 con 0x7f1bb41457f0 2024-03-29T11:26:41.150 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.157+0000 7f1ba9ffb640 1 -- 172.21.15.104:0/3098813976 <== mon.0 v1:172.21.15.104:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1bac003990 con 0x7f1bb41457f0 2024-03-29T11:26:41.150 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.157+0000 7f1ba9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:41.150 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.157+0000 7f1bab7fe640 4 set_mon_vals no callback set 2024-03-29T11:26:41.150 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.157+0000 7f1ba9ffb640 1 -- 172.21.15.104:0/3098813976 <== mon.0 v1:172.21.15.104:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f1bac003de0 con 0x7f1bb41457f0 2024-03-29T11:26:41.150 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.157+0000 7f1ba9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:41.150 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.157+0000 7f1ba9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:41.150 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.157+0000 7f1ba9ffb640 10 monclient: dump: 2024-03-29T11:26:41.150 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:26:41.150 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:26:41.151 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:41.151 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:41.151 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:41.151 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:26:41.151 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:26:41.151 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:26:41.151 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:26:41.151 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:26:41.151 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.159+0000 7f1ba9ffb640 1 -- 172.21.15.104:0/3098813976 <== mon.0 v1:172.21.15.104:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f1bac004000 con 0x7f1bb41457f0 2024-03-29T11:26:41.152 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.160+0000 7f1ba9ffb640 1 -- 172.21.15.104:0/3098813976 <== mon.0 v1:172.21.15.104:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f1bac084df0 con 0x7f1bb41457f0 2024-03-29T11:26:41.152 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.160+0000 7f1ba9ffb640 10 client.4503.objecter ms_dispatch 0x7f1bb4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:41.152 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.160+0000 7f1ba9ffb640 3 client.4503.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:41.152 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.160+0000 7f1ba9ffb640 3 client.4503.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:41.152 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.160+0000 7f1ba9ffb640 20 client.4503.objecter dump_active .. 0 homeless 2024-03-29T11:26:41.153 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.160+0000 7f1bb90c0640 10 monclient: get_auth_request con 0x7f1b9806ddb0 auth_method 0 2024-03-29T11:26:41.153 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.161+0000 7f1ba9ffb640 10 client.4503.objecter ms_handle_connect 0x7f1b9806ddb0 2024-03-29T11:26:41.154 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.167+0000 7f1ba9ffb640 1 -- 172.21.15.104:0/3098813976 <== mon.0 v1:172.21.15.104:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1bac050790 con 0x7f1bb41457f0 2024-03-29T11:26:41.154 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.167+0000 7f1ba9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:41.154 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.167+0000 7f1ba9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:41.498 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.512+0000 7f1bbb324640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:41.498 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.512+0000 7f1bbb324640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:41.499 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.512+0000 7f1bbb324640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:26:41.499 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.512+0000 7f1bbb324640 1 -- 172.21.15.104:0/3098813976 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1bb40b3680 con 0x7f1bb41457f0 2024-03-29T11:26:41.499 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.513+0000 7f1ba9ffb640 1 -- 172.21.15.104:0/3098813976 <== mon.0 v1:172.21.15.104:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f1bac0526c0 con 0x7f1bb41457f0 2024-03-29T11:26:41.499 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.513+0000 7f1ba9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:41.499 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.513+0000 7f1ba9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:41.500 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:26:41.500 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:41.503 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.516+0000 7f1bbb324640 10 librados: watch_flush enter 2024-03-29T11:26:41.503 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.517+0000 7f1bbb324640 10 librados: watch_flush exit 2024-03-29T11:26:41.503 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.517+0000 7f1bbb324640 20 client.4503.objecter shutdown clearing up homeless session... 2024-03-29T11:26:41.503 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.517+0000 7f1bbb324640 10 client.4503.objecter successfully canceled tick 2024-03-29T11:26:41.503 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.517+0000 7f1bbb324640 1 -- 172.21.15.104:0/3098813976 >> v1:172.21.15.104:6816/3341126660 conn(0x7f1b9806ddb0 legacy=0x7f1b98070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:41.503 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.517+0000 7f1bbb324640 10 monclient: shutdown 2024-03-29T11:26:41.503 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.517+0000 7f1bbb324640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:41.504 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.517+0000 7f1bbb324640 1 -- 172.21.15.104:0/3098813976 >> v1:172.21.15.104:6789/0 conn(0x7f1bb41457f0 legacy=0x7f1bb4140790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:41.504 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.517+0000 7f1bbb324640 1 -- 172.21.15.104:0/3098813976 shutdown_connections 2024-03-29T11:26:41.504 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.517+0000 7f1bbb324640 1 -- 172.21.15.104:0/3098813976 >> 172.21.15.104:0/3098813976 conn(0x7f1bb4066a00 msgr2=0x7f1bb40b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:41.504 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.517+0000 7f1bbb324640 1 -- 172.21.15.104:0/3098813976 shutdown_connections 2024-03-29T11:26:41.504 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.517+0000 7f1bbb324640 1 -- 172.21.15.104:0/3098813976 wait complete. 2024-03-29T11:26:41.504 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:41.517+0000 7f1bbb324640 1 librados: shutdown 2024-03-29T11:26:41.517 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:48.325 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:48.473 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.485+0000 7f3c3d58c640 10 monclient: get_monmap_and_config 2024-03-29T11:26:48.473 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.485+0000 7f3c3d58c640 10 monclient: build_initial_monmap 2024-03-29T11:26:48.473 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.485+0000 7f3c3d58c640 10 monclient: monmap: 2024-03-29T11:26:48.473 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:26:48.473 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:26:48.473 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:26:48.487449+0000 2024-03-29T11:26:48.473 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:26:48.487449+0000 2024-03-29T11:26:48.474 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:48.474 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:26:48.474 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:26:48.474 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:26:48.474 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:26:48.474 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:26:48.474 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.487+0000 7f3c3d58c640 1 Processor -- start 2024-03-29T11:26:48.475 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.487+0000 7f3c3d58c640 1 -- start start 2024-03-29T11:26:48.476 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.487+0000 7f3c3d58c640 10 monclient: init 2024-03-29T11:26:48.476 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.487+0000 7f3c3d58c640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:48.476 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.487+0000 7f3c3d58c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:26:48.483 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.488+0000 7f3c3d58c640 10 monclient(hunting): picked mon.noname-c con 0x7f3c381457f0 addr v1:172.21.15.119:6789/0 2024-03-29T11:26:48.483 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.491+0000 7f3c3d58c640 10 monclient(hunting): picked mon.noname-a con 0x7f3c3813f650 addr v1:172.21.15.104:6789/0 2024-03-29T11:26:48.483 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.492+0000 7f3c3d58c640 10 monclient(hunting): picked mon.noname-b con 0x7f3c380a9d70 addr v1:172.21.15.104:6790/0 2024-03-29T11:26:48.484 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.492+0000 7f3c3d58c640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c380aaef0 con 0x7f3c3813f650 2024-03-29T11:26:48.484 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.492+0000 7f3c3d58c640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c38145bd0 con 0x7f3c381457f0 2024-03-29T11:26:48.484 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.492+0000 7f3c3d58c640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c38145eb0 con 0x7f3c380a9d70 2024-03-29T11:26:48.484 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.492+0000 7f3c3d58c640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:48.485 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.492+0000 7f3c3d58c640 10 monclient(hunting): authenticate will time out at 1389.744075s 2024-03-29T11:26:48.485 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.492+0000 7f3c377fe640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f3c380a9d70 0x7f3c3814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:45198/0 (socket says 172.21.15.104:45198) 2024-03-29T11:26:48.485 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.492+0000 7f3c36ffd640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f3c381457f0 0x7f3c3813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:59669/0 (socket says 172.21.15.104:59669) 2024-03-29T11:26:48.487 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.492+0000 7f3c377fe640 1 -- 172.21.15.104:0/4127500093 learned_addr learned my addr 172.21.15.104:0/4127500093 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:26:48.487 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.492+0000 7f3c35ffb640 1 -- 172.21.15.104:0/4127500093 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2727037672 0 0) 0x7f3c38145bd0 con 0x7f3c381457f0 2024-03-29T11:26:48.487 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.492+0000 7f3c35ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:48.487 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.492+0000 7f3c35ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:48.487 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.492+0000 7f3c35ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:48.487 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.492+0000 7f3c35ffb640 10 monclient(hunting): my global_id is 4483 2024-03-29T11:26:48.488 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.492+0000 7f3c35ffb640 1 -- 172.21.15.104:0/4127500093 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c20003520 con 0x7f3c381457f0 2024-03-29T11:26:48.488 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.493+0000 7f3c35ffb640 1 -- 172.21.15.104:0/4127500093 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1638208456 0 0) 0x7f3c38145eb0 con 0x7f3c380a9d70 2024-03-29T11:26:48.488 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.493+0000 7f3c35ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:48.488 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.493+0000 7f3c35ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:48.488 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.493+0000 7f3c35ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:48.488 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.493+0000 7f3c35ffb640 10 monclient(hunting): my global_id is 4514 2024-03-29T11:26:48.488 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.493+0000 7f3c35ffb640 1 -- 172.21.15.104:0/4127500093 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c38145bd0 con 0x7f3c380a9d70 2024-03-29T11:26:48.488 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.493+0000 7f3c35ffb640 1 -- 172.21.15.104:0/4127500093 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2075289106 0 0) 0x7f3c20003520 con 0x7f3c381457f0 2024-03-29T11:26:48.488 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.493+0000 7f3c35ffb640 1 -- 172.21.15.104:0/4127500093 >> v1:172.21.15.104:6790/0 conn(0x7f3c380a9d70 legacy=0x7f3c3814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:48.488 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.493+0000 7f3c35ffb640 1 -- 172.21.15.104:0/4127500093 >> v1:172.21.15.104:6789/0 conn(0x7f3c3813f650 legacy=0x7f3c380b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:26:48.488 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.493+0000 7f3c35ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:26:48.488 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.493+0000 7f3c35ffb640 1 monclient: found mon.noname-c 2024-03-29T11:26:48.488 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.493+0000 7f3c35ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:48.488 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.493+0000 7f3c35ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.119:6789/0 2024-03-29T11:26:48.488 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.493+0000 7f3c35ffb640 1 -- 172.21.15.104:0/4127500093 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c38146340 con 0x7f3c381457f0 2024-03-29T11:26:48.489 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.493+0000 7f3c35ffb640 10 monclient: _finish_auth 0 2024-03-29T11:26:48.489 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.493+0000 7f3c35ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:26:48.489 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.493+0000 7f3c35ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:48.489 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.493+0000 7f3c35ffb640 1 -- 172.21.15.104:0/4127500093 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f3c30001a10 con 0x7f3c381457f0 2024-03-29T11:26:48.489 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.493+0000 7f3c35ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:48.489 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.493+0000 7f3c35ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:48.489 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.493+0000 7f3c35ffb640 10 monclient: dump: 2024-03-29T11:26:48.489 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:26:48.489 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:26:48.489 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:48.489 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:48.489 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:48.489 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:26:48.489 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:26:48.489 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:26:48.490 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:26:48.490 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:26:48.490 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.495+0000 7f3c3d58c640 5 monclient: authenticate success, global_id 4483 2024-03-29T11:26:48.490 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.495+0000 7f3c3d58c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:48.493 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.496+0000 7f3c35ffb640 1 -- 172.21.15.104:0/4127500093 <== mon.1 v1:172.21.15.119:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c30001b70 con 0x7f3c381457f0 2024-03-29T11:26:48.493 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.496+0000 7f3c35ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:48.493 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.496+0000 7f3c35ffb640 1 -- 172.21.15.104:0/4127500093 <== mon.1 v1:172.21.15.119:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f3c30001f80 con 0x7f3c381457f0 2024-03-29T11:26:48.493 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.496+0000 7f3c35ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:48.493 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.496+0000 7f3c35ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:48.493 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.496+0000 7f3c35ffb640 10 monclient: dump: 2024-03-29T11:26:48.493 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:26:48.493 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:26:48.493 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:48.493 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:48.493 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:48.494 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:26:48.494 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:26:48.494 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:26:48.494 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:26:48.494 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:26:48.494 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.496+0000 7f3c3d58c640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:48.494 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.496+0000 7f3c3d58c640 4 set_mon_vals no callback set 2024-03-29T11:26:48.494 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.497+0000 7f3c3d58c640 10 monclient: shutdown 2024-03-29T11:26:48.494 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.497+0000 7f3c3d58c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:48.494 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.497+0000 7f3c3d58c640 1 -- 172.21.15.104:0/4127500093 >> v1:172.21.15.119:6789/0 conn(0x7f3c381457f0 legacy=0x7f3c3813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:48.494 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.498+0000 7f3c3d58c640 1 -- 172.21.15.104:0/4127500093 shutdown_connections 2024-03-29T11:26:48.494 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.498+0000 7f3c3d58c640 1 -- 172.21.15.104:0/4127500093 >> 172.21.15.104:0/4127500093 conn(0x7f3c38066a00 msgr2=0x7f3c38066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:48.494 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.498+0000 7f3c3d58c640 1 -- 172.21.15.104:0/4127500093 shutdown_connections 2024-03-29T11:26:48.494 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.499+0000 7f3c3d58c640 1 -- 172.21.15.104:0/4127500093 wait complete. 2024-03-29T11:26:48.495 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.500+0000 7f3c3d58c640 10 monclient: build_initial_monmap 2024-03-29T11:26:48.495 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.500+0000 7f3c3d58c640 10 monclient: monmap: 2024-03-29T11:26:48.495 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:26:48.495 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:48.495 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:26:48.495 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:26:48.495 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:48.495 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:26:48.495 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:26:48.495 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:26:48.495 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:26:48.495 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:26:48.495 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.501+0000 7f3c3d58c640 1 librados: starting msgr at 2024-03-29T11:26:48.495 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.501+0000 7f3c3d58c640 1 librados: starting objecter 2024-03-29T11:26:48.496 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.501+0000 7f3c3d58c640 1 Processor -- start 2024-03-29T11:26:48.496 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.501+0000 7f3c3d58c640 1 -- start start 2024-03-29T11:26:48.496 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.501+0000 7f3c3d58c640 1 librados: setting wanted keys 2024-03-29T11:26:48.496 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.501+0000 7f3c3d58c640 1 librados: calling monclient init 2024-03-29T11:26:48.496 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.501+0000 7f3c3d58c640 10 monclient: init 2024-03-29T11:26:48.496 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.502+0000 7f3c3d58c640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:48.496 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.502+0000 7f3c3d58c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:26:48.496 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.504+0000 7f3c3d58c640 10 monclient(hunting): picked mon.noname-a con 0x7f3c380a9d70 addr v1:172.21.15.104:6789/0 2024-03-29T11:26:48.496 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.504+0000 7f3c3d58c640 10 monclient(hunting): picked mon.noname-b con 0x7f3c3813f650 addr v1:172.21.15.119:6789/0 2024-03-29T11:26:48.496 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.504+0000 7f3c3d58c640 10 monclient(hunting): picked mon.noname-c con 0x7f3c381457f0 addr v1:172.21.15.104:6790/0 2024-03-29T11:26:48.496 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.504+0000 7f3c3d58c640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c381d6c70 con 0x7f3c380a9d70 2024-03-29T11:26:48.496 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.504+0000 7f3c3d58c640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c3813df10 con 0x7f3c3813f650 2024-03-29T11:26:48.496 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.504+0000 7f3c3d58c640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c38139640 con 0x7f3c381457f0 2024-03-29T11:26:48.496 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.504+0000 7f3c3d58c640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:48.496 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.504+0000 7f3c3d58c640 10 monclient(hunting): authenticate will time out at 1389.756932s 2024-03-29T11:26:48.497 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.505+0000 7f3c377fe640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f3c381457f0 0x7f3c381de8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:46664/0 (socket says 172.21.15.104:46664) 2024-03-29T11:26:48.497 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.505+0000 7f3c377fe640 1 -- 172.21.15.104:0/686413233 learned_addr learned my addr 172.21.15.104:0/686413233 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:26:48.511 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.505+0000 7f3c1f7fe640 10 client.?.objecter ms_handle_connect 0x7f3c381457f0 2024-03-29T11:26:48.511 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.505+0000 7f3c1f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:48.512 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.506+0000 7f3c1f7fe640 1 -- 172.21.15.104:0/686413233 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3806280854 0 0) 0x7f3c38139640 con 0x7f3c381457f0 2024-03-29T11:26:48.512 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.506+0000 7f3c1f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:48.512 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.506+0000 7f3c1f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:48.512 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.506+0000 7f3c1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:48.512 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.506+0000 7f3c1f7fe640 10 monclient(hunting): my global_id is 4517 2024-03-29T11:26:48.512 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.506+0000 7f3c1f7fe640 1 -- 172.21.15.104:0/686413233 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c24004eb0 con 0x7f3c381457f0 2024-03-29T11:26:48.512 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.506+0000 7f3c1f7fe640 10 client.?.objecter ms_handle_connect 0x7f3c380a9d70 2024-03-29T11:26:48.512 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.506+0000 7f3c1f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:48.512 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.508+0000 7f3c1f7fe640 10 client.?.objecter ms_handle_connect 0x7f3c3813f650 2024-03-29T11:26:48.512 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.508+0000 7f3c1f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:48.512 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.508+0000 7f3c1f7fe640 1 -- 172.21.15.104:0/686413233 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3738235508 0 0) 0x7f3c381d6c70 con 0x7f3c380a9d70 2024-03-29T11:26:48.512 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.508+0000 7f3c1f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:48.512 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.508+0000 7f3c1f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:48.512 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.508+0000 7f3c1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:48.513 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.508+0000 7f3c1f7fe640 10 monclient(hunting): my global_id is 4506 2024-03-29T11:26:48.513 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.508+0000 7f3c1f7fe640 1 -- 172.21.15.104:0/686413233 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c38139640 con 0x7f3c380a9d70 2024-03-29T11:26:48.513 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.508+0000 7f3c1f7fe640 1 -- 172.21.15.104:0/686413233 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 574496447 0 0) 0x7f3c3813df10 con 0x7f3c3813f650 2024-03-29T11:26:48.513 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.508+0000 7f3c1f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:48.513 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.508+0000 7f3c1f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:48.513 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.508+0000 7f3c1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:48.513 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.508+0000 7f3c1f7fe640 10 monclient(hunting): my global_id is 4486 2024-03-29T11:26:48.513 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.508+0000 7f3c1f7fe640 1 -- 172.21.15.104:0/686413233 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c381d6c70 con 0x7f3c3813f650 2024-03-29T11:26:48.513 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.508+0000 7f3c1f7fe640 1 -- 172.21.15.104:0/686413233 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 31250830 0 0) 0x7f3c24004eb0 con 0x7f3c381457f0 2024-03-29T11:26:48.513 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.508+0000 7f3c1f7fe640 1 -- 172.21.15.104:0/686413233 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c3813df10 con 0x7f3c381457f0 2024-03-29T11:26:48.513 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.508+0000 7f3c1f7fe640 1 -- 172.21.15.104:0/686413233 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f3c28001930 con 0x7f3c381457f0 2024-03-29T11:26:48.513 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.508+0000 7f3c1f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:48.514 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.508+0000 7f3c1f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:48.514 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.508+0000 7f3c1f7fe640 10 monclient(hunting): dump: 2024-03-29T11:26:48.514 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:26:48.514 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:26:48.514 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:48.514 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:48.514 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:48.514 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:26:48.514 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:26:48.514 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:26:48.514 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:26:48.514 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:26:48.514 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.508+0000 7f3c1f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:48.514 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.508+0000 7f3c1f7fe640 1 -- 172.21.15.104:0/686413233 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4159890024 0 0) 0x7f3c381d6c70 con 0x7f3c3813f650 2024-03-29T11:26:48.515 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.508+0000 7f3c1f7fe640 1 -- 172.21.15.104:0/686413233 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c24004eb0 con 0x7f3c3813f650 2024-03-29T11:26:48.515 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.508+0000 7f3c1f7fe640 1 -- 172.21.15.104:0/686413233 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f3c2c001510 con 0x7f3c3813f650 2024-03-29T11:26:48.515 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.508+0000 7f3c1f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:48.515 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.508+0000 7f3c1f7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:48.515 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.508+0000 7f3c1f7fe640 10 monclient(hunting): dump: 2024-03-29T11:26:48.515 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:26:48.515 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:26:48.515 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:48.515 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:48.515 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:48.515 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:26:48.515 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:26:48.515 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:26:48.515 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:26:48.516 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:26:48.516 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.509+0000 7f3c1f7fe640 1 -- 172.21.15.104:0/686413233 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1106243101 0 0) 0x7f3c24004eb0 con 0x7f3c3813f650 2024-03-29T11:26:48.516 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.509+0000 7f3c1f7fe640 1 -- 172.21.15.104:0/686413233 >> v1:172.21.15.104:6790/0 conn(0x7f3c381457f0 legacy=0x7f3c381de8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:48.516 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.509+0000 7f3c1f7fe640 1 -- 172.21.15.104:0/686413233 >> v1:172.21.15.104:6789/0 conn(0x7f3c380a9d70 legacy=0x7f3c381d6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:48.516 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.509+0000 7f3c1f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:26:48.516 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.509+0000 7f3c1f7fe640 1 monclient: found mon.b 2024-03-29T11:26:48.516 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.509+0000 7f3c1f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:48.516 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.509+0000 7f3c1f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:26:48.516 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.509+0000 7f3c1f7fe640 1 -- 172.21.15.104:0/686413233 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c38139920 con 0x7f3c3813f650 2024-03-29T11:26:48.516 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.509+0000 7f3c1f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:26:48.516 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.509+0000 7f3c1f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:26:48.516 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.509+0000 7f3c1f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:48.516 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.509+0000 7f3c1f7fe640 1 -- 172.21.15.104:0/686413233 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c2c001a00 con 0x7f3c3813f650 2024-03-29T11:26:48.516 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.509+0000 7f3c1f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:48.517 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.509+0000 7f3c1f7fe640 1 -- 172.21.15.104:0/686413233 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f3c2c0038d0 con 0x7f3c3813f650 2024-03-29T11:26:48.517 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.509+0000 7f3c1f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:48.517 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.509+0000 7f3c1f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:48.517 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.509+0000 7f3c1f7fe640 10 monclient: dump: 2024-03-29T11:26:48.517 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:26:48.517 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:26:48.517 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:48.517 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:48.517 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:48.517 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:26:48.517 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:26:48.517 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:26:48.517 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:26:48.517 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:26:48.517 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.509+0000 7f3c3d58c640 5 monclient: authenticate success, global_id 4486 2024-03-29T11:26:48.518 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.509+0000 7f3c3d58c640 10 monclient: _renew_subs 2024-03-29T11:26:48.518 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.509+0000 7f3c3d58c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:26:48.518 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.509+0000 7f3c3d58c640 1 -- 172.21.15.104:0/686413233 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c38144a30 con 0x7f3c3813f650 2024-03-29T11:26:48.518 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.509+0000 7f3c3d58c640 10 client.4486.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:48.518 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.509+0000 7f3c3d58c640 10 monclient: _renew_subs 2024-03-29T11:26:48.518 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.509+0000 7f3c3d58c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:26:48.518 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.509+0000 7f3c3d58c640 1 -- 172.21.15.104:0/686413233 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c381388c0 con 0x7f3c3813f650 2024-03-29T11:26:48.518 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.509+0000 7f3c3d58c640 1 librados: init done 2024-03-29T11:26:48.518 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.510+0000 7f3c357fa640 4 set_mon_vals no callback set 2024-03-29T11:26:48.518 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.511+0000 7f3c3d58c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:48.518 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.511+0000 7f3c3d58c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:48.519 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.511+0000 7f3c3d58c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:26:48.519 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.511+0000 7f3c3d58c640 1 -- 172.21.15.104:0/686413233 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c18005080 con 0x7f3c3813f650 2024-03-29T11:26:48.519 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.512+0000 7f3c1f7fe640 1 -- 172.21.15.104:0/686413233 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f3c2c0198a0 con 0x7f3c3813f650 2024-03-29T11:26:48.519 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.512+0000 7f3c36ffd640 10 monclient: get_auth_request con 0x7f3c2406df50 auth_method 0 2024-03-29T11:26:48.519 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.513+0000 7f3c1f7fe640 1 -- 172.21.15.104:0/686413233 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f3c2c053550 con 0x7f3c3813f650 2024-03-29T11:26:48.519 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.513+0000 7f3c1f7fe640 10 client.4486.objecter ms_dispatch 0x7f3c38000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:48.519 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.513+0000 7f3c1f7fe640 3 client.4486.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:48.519 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.513+0000 7f3c1f7fe640 3 client.4486.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:48.519 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.513+0000 7f3c1f7fe640 20 client.4486.objecter dump_active .. 0 homeless 2024-03-29T11:26:48.519 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.513+0000 7f3c1f7fe640 10 client.4486.objecter ms_handle_connect 0x7f3c2406df50 2024-03-29T11:26:48.519 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.520+0000 7f3c1f7fe640 1 -- 172.21.15.104:0/686413233 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3c2c04f7f0 con 0x7f3c3813f650 2024-03-29T11:26:48.519 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.520+0000 7f3c1f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:48.519 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.520+0000 7f3c1f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:48.832 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.844+0000 7f3c3d58c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:48.832 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.844+0000 7f3c3d58c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:48.832 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.844+0000 7f3c3d58c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:26:48.833 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.844+0000 7f3c3d58c640 1 -- 172.21.15.104:0/686413233 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3c18005330 con 0x7f3c3813f650 2024-03-29T11:26:48.836 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.848+0000 7f3c1f7fe640 1 -- 172.21.15.104:0/686413233 <== mon.1 v1:172.21.15.119:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f3c2c04f990 con 0x7f3c3813f650 2024-03-29T11:26:48.836 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.848+0000 7f3c1f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:48.837 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:26:48.837 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:48.837 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.848+0000 7f3c1f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:48.840 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.851+0000 7f3c3d58c640 10 librados: watch_flush enter 2024-03-29T11:26:48.840 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.852+0000 7f3c3d58c640 10 librados: watch_flush exit 2024-03-29T11:26:48.840 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.852+0000 7f3c3d58c640 20 client.4486.objecter shutdown clearing up homeless session... 2024-03-29T11:26:48.840 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.852+0000 7f3c3d58c640 10 client.4486.objecter successfully canceled tick 2024-03-29T11:26:48.841 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.853+0000 7f3c3d58c640 1 -- 172.21.15.104:0/686413233 >> v1:172.21.15.104:6816/3341126660 conn(0x7f3c2406df50 legacy=0x7f3c24070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:48.841 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.853+0000 7f3c3d58c640 10 monclient: shutdown 2024-03-29T11:26:48.842 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.853+0000 7f3c3d58c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:48.842 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.853+0000 7f3c3d58c640 1 -- 172.21.15.104:0/686413233 >> v1:172.21.15.119:6789/0 conn(0x7f3c3813f650 legacy=0x7f3c38131d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:48.842 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.853+0000 7f3c3d58c640 1 -- 172.21.15.104:0/686413233 shutdown_connections 2024-03-29T11:26:48.842 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.853+0000 7f3c3d58c640 1 -- 172.21.15.104:0/686413233 >> 172.21.15.104:0/686413233 conn(0x7f3c38066a00 msgr2=0x7f3c3813a710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:48.842 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.853+0000 7f3c3d58c640 1 -- 172.21.15.104:0/686413233 shutdown_connections 2024-03-29T11:26:48.842 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.853+0000 7f3c3d58c640 1 -- 172.21.15.104:0/686413233 wait complete. 2024-03-29T11:26:48.843 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:48.853+0000 7f3c3d58c640 1 librados: shutdown 2024-03-29T11:26:48.857 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:55.665 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:55.818 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.830+0000 7fb2d359e640 10 monclient: get_monmap_and_config 2024-03-29T11:26:55.818 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.830+0000 7fb2d359e640 10 monclient: build_initial_monmap 2024-03-29T11:26:55.818 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.830+0000 7fb2d359e640 10 monclient: monmap: 2024-03-29T11:26:55.818 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:26:55.818 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:26:55.818 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:26:55.832039+0000 2024-03-29T11:26:55.818 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:26:55.832039+0000 2024-03-29T11:26:55.818 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:55.818 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:26:55.819 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:26:55.819 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:26:55.819 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:26:55.819 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:26:55.819 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.831+0000 7fb2d359e640 1 Processor -- start 2024-03-29T11:26:55.819 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.831+0000 7fb2d359e640 1 -- start start 2024-03-29T11:26:55.819 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.831+0000 7fb2d359e640 10 monclient: init 2024-03-29T11:26:55.823 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.832+0000 7fb2d359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:55.823 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.832+0000 7fb2d359e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:26:55.823 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.835+0000 7fb2d259c640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fb2d40b2800 0x7fb2d4142620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:60901/0 (socket says 172.21.15.104:60901) 2024-03-29T11:26:55.823 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.835+0000 7fb2d259c640 1 -- 172.21.15.104:0/4039066910 learned_addr learned my addr 172.21.15.104:0/4039066910 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:26:55.823 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.835+0000 7fb2d359e640 10 monclient(hunting): picked mon.noname-a con 0x7fb2d40b2800 addr v1:172.21.15.104:6789/0 2024-03-29T11:26:55.823 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.835+0000 7fb2d359e640 10 monclient(hunting): picked mon.noname-b con 0x7fb2d40a9c00 addr v1:172.21.15.104:6790/0 2024-03-29T11:26:55.825 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.835+0000 7fb2d359e640 10 monclient(hunting): picked mon.noname-c con 0x7fb2d40b38c0 addr v1:172.21.15.119:6789/0 2024-03-29T11:26:55.825 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.836+0000 7fb2d359e640 1 -- 172.21.15.104:0/4039066910 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d4141eb0 con 0x7fb2d40b2800 2024-03-29T11:26:55.826 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.836+0000 7fb2d359e640 1 -- 172.21.15.104:0/4039066910 --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d40b22d0 con 0x7fb2d40b38c0 2024-03-29T11:26:55.826 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.836+0000 7fb2d359e640 1 -- 172.21.15.104:0/4039066910 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d4142d30 con 0x7fb2d40a9c00 2024-03-29T11:26:55.826 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.836+0000 7fb2d359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:55.826 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.836+0000 7fb2d359e640 10 monclient(hunting): authenticate will time out at 1397.088476s 2024-03-29T11:26:55.826 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.836+0000 7fb2d159a640 1 -- 172.21.15.104:0/4039066910 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1434257549 0 0) 0x7fb2d4141eb0 con 0x7fb2d40b2800 2024-03-29T11:26:55.826 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.836+0000 7fb2d159a640 10 monclient(hunting): _negotiate 2024-03-29T11:26:55.826 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.836+0000 7fb2d159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:55.826 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.836+0000 7fb2d159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:55.826 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.836+0000 7fb2d159a640 10 monclient(hunting): my global_id is 4509 2024-03-29T11:26:55.827 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.836+0000 7fb2d159a640 1 -- 172.21.15.104:0/4039066910 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2c0003520 con 0x7fb2d40b2800 2024-03-29T11:26:55.827 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.836+0000 7fb2d159a640 1 -- 172.21.15.104:0/4039066910 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 823022520 0 0) 0x7fb2d4142d30 con 0x7fb2d40a9c00 2024-03-29T11:26:55.827 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.836+0000 7fb2d159a640 10 monclient(hunting): _negotiate 2024-03-29T11:26:55.827 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.836+0000 7fb2d159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:55.827 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.836+0000 7fb2d159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:55.827 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.836+0000 7fb2d159a640 10 monclient(hunting): my global_id is 4520 2024-03-29T11:26:55.827 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.836+0000 7fb2d159a640 1 -- 172.21.15.104:0/4039066910 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2d4141eb0 con 0x7fb2d40a9c00 2024-03-29T11:26:55.827 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.836+0000 7fb2d159a640 1 -- 172.21.15.104:0/4039066910 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1891205686 0 0) 0x7fb2d4141eb0 con 0x7fb2d40a9c00 2024-03-29T11:26:55.827 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.837+0000 7fb2d159a640 1 -- 172.21.15.104:0/4039066910 >> v1:172.21.15.119:6789/0 conn(0x7fb2d40b38c0 legacy=0x7fb2d414d7d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:55.830 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.839+0000 7fb2d159a640 1 -- 172.21.15.104:0/4039066910 >> v1:172.21.15.104:6789/0 conn(0x7fb2d40b2800 legacy=0x7fb2d4142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:55.830 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.839+0000 7fb2d159a640 10 monclient: _finish_hunting 0 2024-03-29T11:26:55.830 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.839+0000 7fb2d159a640 1 monclient: found mon.noname-b 2024-03-29T11:26:55.830 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.839+0000 7fb2d159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:55.830 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.839+0000 7fb2d159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.104:6790/0 2024-03-29T11:26:55.830 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.839+0000 7fb2d159a640 1 -- 172.21.15.104:0/4039066910 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d40aaf90 con 0x7fb2d40a9c00 2024-03-29T11:26:55.830 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.839+0000 7fb2d159a640 10 monclient: _finish_auth 0 2024-03-29T11:26:55.831 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.839+0000 7fb2d159a640 10 monclient: _check_auth_tickets 2024-03-29T11:26:55.831 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.839+0000 7fb2d159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:55.831 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.839+0000 7fb2d159a640 1 -- 172.21.15.104:0/4039066910 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fb2c4001670 con 0x7fb2d40a9c00 2024-03-29T11:26:55.831 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.840+0000 7fb2d359e640 5 monclient: authenticate success, global_id 4520 2024-03-29T11:26:55.831 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.840+0000 7fb2d359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:55.831 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.840+0000 7fb2d159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:55.831 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.840+0000 7fb2d159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:55.831 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.840+0000 7fb2d159a640 10 monclient: dump: 2024-03-29T11:26:55.831 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:26:55.833 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:26:55.834 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:55.834 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:55.834 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:55.834 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:26:55.834 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:26:55.834 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:26:55.835 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:26:55.835 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:26:55.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.840+0000 7fb2d159a640 1 -- 172.21.15.104:0/4039066910 <== mon.2 v1:172.21.15.104:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2c4001890 con 0x7fb2d40a9c00 2024-03-29T11:26:55.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.840+0000 7fb2d159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:55.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.840+0000 7fb2d159a640 1 -- 172.21.15.104:0/4039066910 <== mon.2 v1:172.21.15.104:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fb2c4001ba0 con 0x7fb2d40a9c00 2024-03-29T11:26:55.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.840+0000 7fb2d159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:55.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.840+0000 7fb2d159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:55.835 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.840+0000 7fb2d159a640 10 monclient: dump: 2024-03-29T11:26:55.835 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:26:55.836 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:26:55.836 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:55.836 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:55.836 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:55.836 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:26:55.836 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:26:55.836 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:26:55.836 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:26:55.836 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:26:55.836 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.840+0000 7fb2d359e640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:55.836 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.840+0000 7fb2d359e640 4 set_mon_vals no callback set 2024-03-29T11:26:55.836 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.841+0000 7fb2d359e640 10 monclient: shutdown 2024-03-29T11:26:55.836 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.841+0000 7fb2d359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:55.836 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.841+0000 7fb2d359e640 1 -- 172.21.15.104:0/4039066910 >> v1:172.21.15.104:6790/0 conn(0x7fb2d40a9c00 legacy=0x7fb2d413bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:55.839 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.843+0000 7fb2d359e640 1 -- 172.21.15.104:0/4039066910 shutdown_connections 2024-03-29T11:26:55.839 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.843+0000 7fb2d359e640 1 -- 172.21.15.104:0/4039066910 >> 172.21.15.104:0/4039066910 conn(0x7fb2d4066a00 msgr2=0x7fb2d4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:55.840 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.844+0000 7fb2d359e640 1 -- 172.21.15.104:0/4039066910 shutdown_connections 2024-03-29T11:26:55.840 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.846+0000 7fb2d359e640 1 -- 172.21.15.104:0/4039066910 wait complete. 2024-03-29T11:26:55.840 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.847+0000 7fb2d359e640 10 monclient: build_initial_monmap 2024-03-29T11:26:55.840 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.847+0000 7fb2d359e640 10 monclient: monmap: 2024-03-29T11:26:55.840 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:26:55.840 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:55.840 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:26:55.840 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:26:55.840 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:55.840 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:26:55.840 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:26:55.840 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:26:55.841 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:26:55.841 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:26:55.841 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.847+0000 7fb2d359e640 1 librados: starting msgr at 2024-03-29T11:26:55.841 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.847+0000 7fb2d359e640 1 librados: starting objecter 2024-03-29T11:26:55.841 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.847+0000 7fb2d359e640 1 Processor -- start 2024-03-29T11:26:55.841 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.847+0000 7fb2d359e640 1 -- start start 2024-03-29T11:26:55.841 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.847+0000 7fb2d359e640 1 librados: setting wanted keys 2024-03-29T11:26:55.841 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.847+0000 7fb2d359e640 1 librados: calling monclient init 2024-03-29T11:26:55.841 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.847+0000 7fb2d359e640 10 monclient: init 2024-03-29T11:26:55.841 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.848+0000 7fb2d359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:55.841 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.848+0000 7fb2d359e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:26:55.841 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.848+0000 7fb2d359e640 10 monclient(hunting): picked mon.noname-a con 0x7fb2d40a9c00 addr v1:172.21.15.104:6789/0 2024-03-29T11:26:55.841 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.848+0000 7fb2d359e640 10 monclient(hunting): picked mon.noname-c con 0x7fb2d40b2800 addr v1:172.21.15.104:6790/0 2024-03-29T11:26:55.841 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.848+0000 7fb2d359e640 10 monclient(hunting): picked mon.noname-b con 0x7fb2d40b38c0 addr v1:172.21.15.119:6789/0 2024-03-29T11:26:55.842 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.848+0000 7fb2d359e640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d413b7d0 con 0x7fb2d40a9c00 2024-03-29T11:26:55.842 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.848+0000 7fb2d359e640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d41f5790 con 0x7fb2d40b38c0 2024-03-29T11:26:55.842 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.848+0000 7fb2d359e640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d40703a0 con 0x7fb2d40b2800 2024-03-29T11:26:55.842 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.848+0000 7fb2d359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:55.842 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.848+0000 7fb2d359e640 10 monclient(hunting): authenticate will time out at 1397.100850s 2024-03-29T11:26:55.842 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.849+0000 7fb2d1d9b640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7fb2d40b2800 0x7fb2d413a9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:54662/0 (socket says 172.21.15.104:54662) 2024-03-29T11:26:55.842 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.849+0000 7fb2d1d9b640 1 -- 172.21.15.104:0/1306980073 learned_addr learned my addr 172.21.15.104:0/1306980073 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:26:55.842 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.849+0000 7fb2caffd640 10 client.?.objecter ms_handle_connect 0x7fb2d40b2800 2024-03-29T11:26:55.842 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.849+0000 7fb2caffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:55.844 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.849+0000 7fb2caffd640 1 -- 172.21.15.104:0/1306980073 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 51093618 0 0) 0x7fb2d40703a0 con 0x7fb2d40b2800 2024-03-29T11:26:55.844 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.849+0000 7fb2caffd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:55.844 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.849+0000 7fb2caffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:55.844 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.849+0000 7fb2caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:55.844 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.849+0000 7fb2caffd640 10 monclient(hunting): my global_id is 4523 2024-03-29T11:26:55.844 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.849+0000 7fb2caffd640 1 -- 172.21.15.104:0/1306980073 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2b4004eb0 con 0x7fb2d40b2800 2024-03-29T11:26:55.845 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.850+0000 7fb2caffd640 1 -- 172.21.15.104:0/1306980073 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4148954589 0 0) 0x7fb2b4004eb0 con 0x7fb2d40b2800 2024-03-29T11:26:55.845 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.850+0000 7fb2caffd640 1 -- 172.21.15.104:0/1306980073 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2d40703a0 con 0x7fb2d40b2800 2024-03-29T11:26:55.845 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.850+0000 7fb2caffd640 1 -- 172.21.15.104:0/1306980073 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fb2c4001630 con 0x7fb2d40b2800 2024-03-29T11:26:55.845 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.850+0000 7fb2caffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:55.845 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.850+0000 7fb2caffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:55.845 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.850+0000 7fb2caffd640 10 monclient(hunting): dump: 2024-03-29T11:26:55.845 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:26:55.845 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:26:55.845 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:55.845 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:55.845 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:55.845 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:26:55.845 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:26:55.845 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:26:55.845 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:26:55.846 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:26:55.846 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.850+0000 7fb2caffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:55.846 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.850+0000 7fb2caffd640 1 -- 172.21.15.104:0/1306980073 <== mon.2 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1910644771 0 0) 0x7fb2d40703a0 con 0x7fb2d40b2800 2024-03-29T11:26:55.846 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.850+0000 7fb2caffd640 1 -- 172.21.15.104:0/1306980073 >> v1:172.21.15.119:6789/0 conn(0x7fb2d40b38c0 legacy=0x7fb2d413b0c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:26:55.849 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.852+0000 7fb2caffd640 1 -- 172.21.15.104:0/1306980073 >> v1:172.21.15.104:6789/0 conn(0x7fb2d40a9c00 legacy=0x7fb2d41f5080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:26:55.850 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.852+0000 7fb2caffd640 10 monclient: _finish_hunting 0 2024-03-29T11:26:55.850 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.852+0000 7fb2caffd640 1 monclient: found mon.c 2024-03-29T11:26:55.850 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.852+0000 7fb2caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:55.850 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.852+0000 7fb2caffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:26:55.850 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.852+0000 7fb2caffd640 1 -- 172.21.15.104:0/1306980073 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d4070530 con 0x7fb2d40b2800 2024-03-29T11:26:55.850 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.852+0000 7fb2caffd640 10 monclient: _finish_auth 0 2024-03-29T11:26:55.850 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.852+0000 7fb2caffd640 10 monclient: _check_auth_tickets 2024-03-29T11:26:55.850 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.852+0000 7fb2caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:55.850 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.852+0000 7fb2d359e640 5 monclient: authenticate success, global_id 4523 2024-03-29T11:26:55.850 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.852+0000 7fb2d359e640 10 monclient: _renew_subs 2024-03-29T11:26:55.850 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.852+0000 7fb2d359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:26:55.850 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.852+0000 7fb2d359e640 1 -- 172.21.15.104:0/1306980073 --> v1:172.21.15.104:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2d406de90 con 0x7fb2d40b2800 2024-03-29T11:26:55.850 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.852+0000 7fb2d359e640 10 client.4523.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:55.851 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.852+0000 7fb2d359e640 10 monclient: _renew_subs 2024-03-29T11:26:55.851 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.852+0000 7fb2d359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:26:55.851 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.852+0000 7fb2d359e640 1 -- 172.21.15.104:0/1306980073 --> v1:172.21.15.104:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2d4071330 con 0x7fb2d40b2800 2024-03-29T11:26:55.851 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.852+0000 7fb2d359e640 1 librados: init done 2024-03-29T11:26:55.851 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.854+0000 7fb2d359e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:55.851 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.854+0000 7fb2d359e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:55.851 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.854+0000 7fb2d359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:26:55.851 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.854+0000 7fb2d359e640 1 -- 172.21.15.104:0/1306980073 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2d4071950 con 0x7fb2d40b2800 2024-03-29T11:26:55.851 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.854+0000 7fb2caffd640 1 -- 172.21.15.104:0/1306980073 <== mon.2 v1:172.21.15.104:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2c40038a0 con 0x7fb2d40b2800 2024-03-29T11:26:55.851 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.854+0000 7fb2caffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:55.851 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.854+0000 7fb2caffd640 1 -- 172.21.15.104:0/1306980073 <== mon.2 v1:172.21.15.104:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fb2c4003cf0 con 0x7fb2d40b2800 2024-03-29T11:26:55.851 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.854+0000 7fb2caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:55.851 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.854+0000 7fb2caffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:55.851 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.854+0000 7fb2caffd640 10 monclient: dump: 2024-03-29T11:26:55.851 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:26:55.852 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:26:55.852 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:55.852 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:26:55.852 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:55.852 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:26:55.852 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:26:55.852 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:26:55.852 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:26:55.852 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:26:55.852 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.854+0000 7fb2d0d99640 4 set_mon_vals no callback set 2024-03-29T11:26:55.852 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.856+0000 7fb2caffd640 1 -- 172.21.15.104:0/1306980073 <== mon.2 v1:172.21.15.104:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7fb2c4003e50 con 0x7fb2d40b2800 2024-03-29T11:26:55.852 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.856+0000 7fb2caffd640 1 -- 172.21.15.104:0/1306980073 <== mon.2 v1:172.21.15.104:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7fb2c4084b70 con 0x7fb2d40b2800 2024-03-29T11:26:55.852 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.856+0000 7fb2caffd640 10 client.4523.objecter ms_dispatch 0x7fb2d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:55.852 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.856+0000 7fb2caffd640 3 client.4523.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:55.852 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.856+0000 7fb2caffd640 3 client.4523.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:55.853 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.856+0000 7fb2d259c640 10 monclient: get_auth_request con 0x7fb2b406dc50 auth_method 0 2024-03-29T11:26:55.853 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.857+0000 7fb2caffd640 20 client.4523.objecter dump_active .. 0 homeless 2024-03-29T11:26:55.853 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.857+0000 7fb2caffd640 10 client.4523.objecter ms_handle_connect 0x7fb2b406dc50 2024-03-29T11:26:55.855 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.865+0000 7fb2caffd640 1 -- 172.21.15.104:0/1306980073 <== mon.2 v1:172.21.15.104:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb2c40504b0 con 0x7fb2d40b2800 2024-03-29T11:26:55.855 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.865+0000 7fb2caffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:55.855 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:55.865+0000 7fb2caffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:56.184 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:56.195+0000 7fb2d359e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:56.184 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:56.195+0000 7fb2d359e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:56.184 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:56.195+0000 7fb2d359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:26:56.184 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:56.195+0000 7fb2d359e640 1 -- 172.21.15.104:0/1306980073 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2d4071680 con 0x7fb2d40b2800 2024-03-29T11:26:56.185 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:56.197+0000 7fb2caffd640 1 -- 172.21.15.104:0/1306980073 <== mon.2 v1:172.21.15.104:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7fb2c40523e0 con 0x7fb2d40b2800 2024-03-29T11:26:56.185 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:56.197+0000 7fb2caffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:56.185 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:56.197+0000 7fb2caffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:56.187 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:26:56.187 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:56.190 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:56.202+0000 7fb2d359e640 10 librados: watch_flush enter 2024-03-29T11:26:56.190 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:56.202+0000 7fb2d359e640 10 librados: watch_flush exit 2024-03-29T11:26:56.190 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:56.202+0000 7fb2d359e640 20 client.4523.objecter shutdown clearing up homeless session... 2024-03-29T11:26:56.190 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:56.202+0000 7fb2d359e640 10 client.4523.objecter successfully canceled tick 2024-03-29T11:26:56.193 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:56.203+0000 7fb2d359e640 1 -- 172.21.15.104:0/1306980073 >> v1:172.21.15.104:6816/3341126660 conn(0x7fb2b406dc50 legacy=0x7fb2b4070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:56.193 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:56.203+0000 7fb2d359e640 10 monclient: shutdown 2024-03-29T11:26:56.193 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:56.203+0000 7fb2d359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:56.193 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:56.203+0000 7fb2d359e640 1 -- 172.21.15.104:0/1306980073 >> v1:172.21.15.104:6790/0 conn(0x7fb2d40b2800 legacy=0x7fb2d413a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:56.193 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:56.204+0000 7fb2d359e640 1 -- 172.21.15.104:0/1306980073 shutdown_connections 2024-03-29T11:26:56.193 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:56.204+0000 7fb2d359e640 1 -- 172.21.15.104:0/1306980073 >> 172.21.15.104:0/1306980073 conn(0x7fb2d4066a00 msgr2=0x7fb2d4140450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:56.194 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:56.204+0000 7fb2d359e640 1 -- 172.21.15.104:0/1306980073 shutdown_connections 2024-03-29T11:26:56.194 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:56.204+0000 7fb2d359e640 1 -- 172.21.15.104:0/1306980073 wait complete. 2024-03-29T11:26:56.194 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:26:56.204+0000 7fb2d359e640 1 librados: shutdown 2024-03-29T11:26:56.210 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:03.019 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:03.166 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.176+0000 7f929d73b640 10 monclient: get_monmap_and_config 2024-03-29T11:27:03.166 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.176+0000 7f929d73b640 10 monclient: build_initial_monmap 2024-03-29T11:27:03.166 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.176+0000 7f929d73b640 10 monclient: monmap: 2024-03-29T11:27:03.166 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:27:03.166 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:27:03.166 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:27:03.177200+0000 2024-03-29T11:27:03.166 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:27:03.177200+0000 2024-03-29T11:27:03.166 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:03.166 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:03.166 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:27:03.166 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:27:03.167 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:27:03.167 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:03.167 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.177+0000 7f929d73b640 1 Processor -- start 2024-03-29T11:27:03.167 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.177+0000 7f929d73b640 1 -- start start 2024-03-29T11:27:03.167 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.177+0000 7f929d73b640 10 monclient: init 2024-03-29T11:27:03.167 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.178+0000 7f929d73b640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:03.167 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.178+0000 7f929d73b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:27:03.167 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.178+0000 7f929d73b640 10 monclient(hunting): picked mon.noname-a con 0x7f92980a9750 addr v1:172.21.15.104:6789/0 2024-03-29T11:27:03.167 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.178+0000 7f929d73b640 10 monclient(hunting): picked mon.noname-c con 0x7f9298149ac0 addr v1:172.21.15.119:6789/0 2024-03-29T11:27:03.167 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.178+0000 7f929d73b640 10 monclient(hunting): picked mon.noname-b con 0x7f929814fe50 addr v1:172.21.15.104:6790/0 2024-03-29T11:27:03.167 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.178+0000 7f929d73b640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92980aa4a0 con 0x7f92980a9750 2024-03-29T11:27:03.167 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.178+0000 7f929d73b640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f929814a680 con 0x7f9298149ac0 2024-03-29T11:27:03.167 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.178+0000 7f929d73b640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92980af290 con 0x7f929814fe50 2024-03-29T11:27:03.168 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.178+0000 7f929d73b640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:03.168 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.178+0000 7f929d73b640 10 monclient(hunting): authenticate will time out at 1404.429187s 2024-03-29T11:27:03.168 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.178+0000 7f92977fe640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f929814fe50 0x7f9298152240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:42222/0 (socket says 172.21.15.104:42222) 2024-03-29T11:27:03.168 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.178+0000 7f92977fe640 1 -- 172.21.15.104:0/3076018423 learned_addr learned my addr 172.21.15.104:0/3076018423 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:27:03.175 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.179+0000 7f9295ffb640 1 -- 172.21.15.104:0/3076018423 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4224302995 0 0) 0x7f929814a680 con 0x7f9298149ac0 2024-03-29T11:27:03.175 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.179+0000 7f9295ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:03.175 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.179+0000 7f9295ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:03.175 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.179+0000 7f9295ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:03.175 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.179+0000 7f9295ffb640 10 monclient(hunting): my global_id is 4489 2024-03-29T11:27:03.175 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.179+0000 7f9295ffb640 1 -- 172.21.15.104:0/3076018423 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9280003520 con 0x7f9298149ac0 2024-03-29T11:27:03.175 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.179+0000 7f9295ffb640 1 -- 172.21.15.104:0/3076018423 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3284626008 0 0) 0x7f92980af290 con 0x7f929814fe50 2024-03-29T11:27:03.175 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.179+0000 7f9295ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:03.175 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.179+0000 7f9295ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:03.175 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.179+0000 7f9295ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:03.176 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.179+0000 7f9295ffb640 10 monclient(hunting): my global_id is 4526 2024-03-29T11:27:03.176 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.179+0000 7f9295ffb640 1 -- 172.21.15.104:0/3076018423 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f929814a680 con 0x7f929814fe50 2024-03-29T11:27:03.176 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.179+0000 7f9295ffb640 1 -- 172.21.15.104:0/3076018423 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 296937899 0 0) 0x7f92980aa4a0 con 0x7f92980a9750 2024-03-29T11:27:03.176 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.179+0000 7f9295ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:03.176 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.179+0000 7f9295ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:03.176 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.179+0000 7f9295ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:03.176 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.179+0000 7f9295ffb640 10 monclient(hunting): my global_id is 4512 2024-03-29T11:27:03.176 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.179+0000 7f9295ffb640 1 -- 172.21.15.104:0/3076018423 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92980af290 con 0x7f92980a9750 2024-03-29T11:27:03.176 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.179+0000 7f9295ffb640 1 -- 172.21.15.104:0/3076018423 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1071211804 0 0) 0x7f929814a680 con 0x7f929814fe50 2024-03-29T11:27:03.176 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.179+0000 7f9295ffb640 1 -- 172.21.15.104:0/3076018423 >> v1:172.21.15.119:6789/0 conn(0x7f9298149ac0 legacy=0x7f9298149f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:03.176 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.179+0000 7f9295ffb640 1 -- 172.21.15.104:0/3076018423 >> v1:172.21.15.104:6789/0 conn(0x7f92980a9750 legacy=0x7f92981436f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:03.176 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.179+0000 7f9295ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:27:03.176 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.179+0000 7f9295ffb640 1 monclient: found mon.noname-b 2024-03-29T11:27:03.176 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.179+0000 7f9295ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:03.177 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.179+0000 7f9295ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.104:6790/0 2024-03-29T11:27:03.177 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.179+0000 7f9295ffb640 1 -- 172.21.15.104:0/3076018423 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92980afb40 con 0x7f929814fe50 2024-03-29T11:27:03.177 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.179+0000 7f9295ffb640 10 monclient: _finish_auth 0 2024-03-29T11:27:03.177 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.179+0000 7f9295ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:27:03.177 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.179+0000 7f9295ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:03.177 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.179+0000 7f929d73b640 5 monclient: authenticate success, global_id 4526 2024-03-29T11:27:03.177 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.179+0000 7f929d73b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:03.177 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.179+0000 7f9295ffb640 1 -- 172.21.15.104:0/3076018423 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f92900019f0 con 0x7f929814fe50 2024-03-29T11:27:03.177 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.179+0000 7f9295ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:03.177 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.179+0000 7f9295ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:03.177 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.179+0000 7f9295ffb640 10 monclient: dump: 2024-03-29T11:27:03.177 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:27:03.177 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:27:03.177 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:03.178 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:03.178 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:03.178 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:03.178 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:27:03.178 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:27:03.178 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:27:03.178 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:03.178 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.179+0000 7f929d73b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:03.178 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.180+0000 7f9295ffb640 1 -- 172.21.15.104:0/3076018423 <== mon.2 v1:172.21.15.104:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9290001c40 con 0x7f929814fe50 2024-03-29T11:27:03.178 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.180+0000 7f9295ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:03.179 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.180+0000 7f9295ffb640 1 -- 172.21.15.104:0/3076018423 <== mon.2 v1:172.21.15.104:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f9290001fd0 con 0x7f929814fe50 2024-03-29T11:27:03.179 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.180+0000 7f9295ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:03.179 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.180+0000 7f9295ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:03.179 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.180+0000 7f9295ffb640 10 monclient: dump: 2024-03-29T11:27:03.179 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:27:03.179 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:27:03.179 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:03.179 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:03.179 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:03.179 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:03.179 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:27:03.179 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:27:03.179 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:27:03.179 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:03.180 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.180+0000 7f929d73b640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:03.180 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.180+0000 7f929d73b640 4 set_mon_vals no callback set 2024-03-29T11:27:03.180 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.181+0000 7f929d73b640 10 monclient: shutdown 2024-03-29T11:27:03.180 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.181+0000 7f929d73b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:03.180 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.181+0000 7f929d73b640 1 -- 172.21.15.104:0/3076018423 >> v1:172.21.15.104:6790/0 conn(0x7f929814fe50 legacy=0x7f9298152240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:03.180 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.181+0000 7f929d73b640 1 -- 172.21.15.104:0/3076018423 shutdown_connections 2024-03-29T11:27:03.180 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.181+0000 7f929d73b640 1 -- 172.21.15.104:0/3076018423 >> 172.21.15.104:0/3076018423 conn(0x7f92980b35e0 msgr2=0x7f92980b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:03.180 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.181+0000 7f929d73b640 1 -- 172.21.15.104:0/3076018423 shutdown_connections 2024-03-29T11:27:03.180 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.181+0000 7f929d73b640 1 -- 172.21.15.104:0/3076018423 wait complete. 2024-03-29T11:27:03.180 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.182+0000 7f929d73b640 10 monclient: build_initial_monmap 2024-03-29T11:27:03.180 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.182+0000 7f929d73b640 10 monclient: monmap: 2024-03-29T11:27:03.180 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:27:03.180 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:03.180 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:27:03.180 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:27:03.181 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:03.181 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:03.181 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:27:03.181 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:27:03.181 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:27:03.181 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:03.181 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.182+0000 7f929d73b640 1 librados: starting msgr at 2024-03-29T11:27:03.181 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.182+0000 7f929d73b640 1 librados: starting objecter 2024-03-29T11:27:03.181 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.183+0000 7f929d73b640 1 Processor -- start 2024-03-29T11:27:03.181 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.183+0000 7f929d73b640 1 -- start start 2024-03-29T11:27:03.181 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.183+0000 7f929d73b640 1 librados: setting wanted keys 2024-03-29T11:27:03.181 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.183+0000 7f929d73b640 1 librados: calling monclient init 2024-03-29T11:27:03.181 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.183+0000 7f929d73b640 10 monclient: init 2024-03-29T11:27:03.181 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.183+0000 7f929d73b640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:03.181 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.183+0000 7f929d73b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:27:03.182 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.183+0000 7f929d73b640 10 monclient(hunting): picked mon.noname-a con 0x7f92980a9750 addr v1:172.21.15.104:6789/0 2024-03-29T11:27:03.182 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.184+0000 7f929d73b640 10 monclient(hunting): picked mon.noname-b con 0x7f9298149ac0 addr v1:172.21.15.119:6789/0 2024-03-29T11:27:03.182 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.184+0000 7f929d73b640 10 monclient(hunting): picked mon.noname-c con 0x7f929814fe50 addr v1:172.21.15.104:6790/0 2024-03-29T11:27:03.182 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.184+0000 7f929d73b640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92980ad370 con 0x7f92980a9750 2024-03-29T11:27:03.182 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.184+0000 7f929d73b640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92980aced0 con 0x7f9298149ac0 2024-03-29T11:27:03.182 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.184+0000 7f929d73b640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92981db900 con 0x7f929814fe50 2024-03-29T11:27:03.182 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.184+0000 7f929d73b640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:03.182 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.184+0000 7f929d73b640 10 monclient(hunting): authenticate will time out at 1404.435101s 2024-03-29T11:27:03.182 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.184+0000 7f9296ffd640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f92980a9750 0x7f92980ac7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:49810/0 (socket says 172.21.15.104:49810) 2024-03-29T11:27:03.182 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.184+0000 7f9296ffd640 1 -- 172.21.15.104:0/2838881127 learned_addr learned my addr 172.21.15.104:0/2838881127 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:27:03.182 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.184+0000 7f927f7fe640 10 client.?.objecter ms_handle_connect 0x7f92980a9750 2024-03-29T11:27:03.183 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.184+0000 7f927f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:03.183 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.185+0000 7f927f7fe640 10 client.?.objecter ms_handle_connect 0x7f9298149ac0 2024-03-29T11:27:03.183 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.185+0000 7f927f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:03.183 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.185+0000 7f927f7fe640 1 -- 172.21.15.104:0/2838881127 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3363979600 0 0) 0x7f92980ad370 con 0x7f92980a9750 2024-03-29T11:27:03.183 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.185+0000 7f927f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:03.183 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.185+0000 7f927f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:03.183 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.185+0000 7f927f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:03.183 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.185+0000 7f927f7fe640 10 monclient(hunting): my global_id is 4515 2024-03-29T11:27:03.183 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.185+0000 7f927f7fe640 1 -- 172.21.15.104:0/2838881127 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9284004eb0 con 0x7f92980a9750 2024-03-29T11:27:03.183 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.185+0000 7f927f7fe640 1 -- 172.21.15.104:0/2838881127 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1053891130 0 0) 0x7f92980aced0 con 0x7f9298149ac0 2024-03-29T11:27:03.183 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.185+0000 7f927f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:03.183 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.185+0000 7f927f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:03.183 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.185+0000 7f927f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:03.184 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.185+0000 7f927f7fe640 10 monclient(hunting): my global_id is 4492 2024-03-29T11:27:03.184 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.185+0000 7f927f7fe640 1 -- 172.21.15.104:0/2838881127 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92980ad370 con 0x7f9298149ac0 2024-03-29T11:27:03.184 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.186+0000 7f927f7fe640 1 -- 172.21.15.104:0/2838881127 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3626064739 0 0) 0x7f92980ad370 con 0x7f9298149ac0 2024-03-29T11:27:03.184 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.186+0000 7f927f7fe640 1 -- 172.21.15.104:0/2838881127 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92980aced0 con 0x7f9298149ac0 2024-03-29T11:27:03.184 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.186+0000 7f927f7fe640 1 -- 172.21.15.104:0/2838881127 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f928c001800 con 0x7f9298149ac0 2024-03-29T11:27:03.184 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.186+0000 7f927f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:03.184 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.186+0000 7f927f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:03.184 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.186+0000 7f927f7fe640 10 monclient(hunting): dump: 2024-03-29T11:27:03.184 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:27:03.184 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:27:03.184 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:03.184 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:03.184 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:03.184 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:03.184 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:27:03.185 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:27:03.185 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:27:03.185 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:03.185 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.186+0000 7f927f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:03.185 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.186+0000 7f927f7fe640 1 -- 172.21.15.104:0/2838881127 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2906056309 0 0) 0x7f9284004eb0 con 0x7f92980a9750 2024-03-29T11:27:03.185 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.186+0000 7f927f7fe640 1 -- 172.21.15.104:0/2838881127 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92980ad370 con 0x7f92980a9750 2024-03-29T11:27:03.185 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.186+0000 7f927f7fe640 1 -- 172.21.15.104:0/2838881127 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f9288001970 con 0x7f92980a9750 2024-03-29T11:27:03.185 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.186+0000 7f927f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:03.185 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.186+0000 7f927f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:03.185 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.186+0000 7f927f7fe640 10 monclient(hunting): dump: 2024-03-29T11:27:03.185 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:27:03.185 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:27:03.185 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:03.185 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:03.186 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:03.186 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:03.186 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:27:03.186 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:27:03.186 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:27:03.186 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:03.186 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.187+0000 7f927f7fe640 10 client.?.objecter ms_handle_connect 0x7f929814fe50 2024-03-29T11:27:03.186 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.187+0000 7f927f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:03.186 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.187+0000 7f927f7fe640 1 -- 172.21.15.104:0/2838881127 <== mon.0 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 357961192 0 0) 0x7f92980ad370 con 0x7f92980a9750 2024-03-29T11:27:03.186 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.187+0000 7f927f7fe640 1 -- 172.21.15.104:0/2838881127 >> v1:172.21.15.104:6790/0 conn(0x7f929814fe50 legacy=0x7f92981e6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:03.186 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.187+0000 7f927f7fe640 1 -- 172.21.15.104:0/2838881127 >> v1:172.21.15.119:6789/0 conn(0x7f9298149ac0 legacy=0x7f92981dbe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:03.186 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.187+0000 7f927f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:27:03.186 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.187+0000 7f927f7fe640 1 monclient: found mon.a 2024-03-29T11:27:03.186 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.187+0000 7f927f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:03.187 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.187+0000 7f927f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:27:03.187 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.187+0000 7f927f7fe640 1 -- 172.21.15.104:0/2838881127 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92981dc530 con 0x7f92980a9750 2024-03-29T11:27:03.187 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.187+0000 7f927f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:27:03.187 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.187+0000 7f927f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:27:03.187 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.187+0000 7f927f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:03.187 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.187+0000 7f929d73b640 5 monclient: authenticate success, global_id 4515 2024-03-29T11:27:03.187 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.187+0000 7f929d73b640 10 monclient: _renew_subs 2024-03-29T11:27:03.187 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.187+0000 7f929d73b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:27:03.187 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.187+0000 7f929d73b640 1 -- 172.21.15.104:0/2838881127 --> v1:172.21.15.104:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9298147930 con 0x7f92980a9750 2024-03-29T11:27:03.187 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.187+0000 7f927f7fe640 1 -- 172.21.15.104:0/2838881127 <== mon.0 v1:172.21.15.104:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9288003610 con 0x7f92980a9750 2024-03-29T11:27:03.187 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.187+0000 7f927f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:03.187 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.187+0000 7f927f7fe640 1 -- 172.21.15.104:0/2838881127 <== mon.0 v1:172.21.15.104:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f9288003a20 con 0x7f92980a9750 2024-03-29T11:27:03.187 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.187+0000 7f927f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:03.187 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.187+0000 7f927f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:03.187 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.187+0000 7f927f7fe640 10 monclient: dump: 2024-03-29T11:27:03.188 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:27:03.188 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:27:03.188 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:03.188 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:03.188 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:03.188 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:03.188 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:27:03.188 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:27:03.188 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:27:03.188 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:03.188 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.187+0000 7f929d73b640 10 client.4515.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:03.188 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.187+0000 7f929d73b640 10 monclient: _renew_subs 2024-03-29T11:27:03.188 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.187+0000 7f929d73b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:27:03.189 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.187+0000 7f929d73b640 1 -- 172.21.15.104:0/2838881127 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f92980abc00 con 0x7f92980a9750 2024-03-29T11:27:03.189 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.187+0000 7f929d73b640 1 librados: init done 2024-03-29T11:27:03.189 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.187+0000 7f92957fa640 4 set_mon_vals no callback set 2024-03-29T11:27:03.190 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.188+0000 7f929d73b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:03.190 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.188+0000 7f929d73b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:03.190 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.188+0000 7f929d73b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:27:03.191 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.188+0000 7f929d73b640 1 -- 172.21.15.104:0/2838881127 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92981dba90 con 0x7f92980a9750 2024-03-29T11:27:03.191 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.189+0000 7f927f7fe640 1 -- 172.21.15.104:0/2838881127 <== mon.0 v1:172.21.15.104:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f9288003c00 con 0x7f92980a9750 2024-03-29T11:27:03.191 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.191+0000 7f92967fc640 10 monclient: get_auth_request con 0x7f928406dee0 auth_method 0 2024-03-29T11:27:03.191 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.191+0000 7f927f7fe640 1 -- 172.21.15.104:0/2838881127 <== mon.0 v1:172.21.15.104:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f9288084b10 con 0x7f92980a9750 2024-03-29T11:27:03.191 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.191+0000 7f927f7fe640 10 client.4515.objecter ms_dispatch 0x7f9298000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:03.191 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.191+0000 7f927f7fe640 3 client.4515.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:03.191 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.191+0000 7f927f7fe640 3 client.4515.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:03.191 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.191+0000 7f927f7fe640 20 client.4515.objecter dump_active .. 0 homeless 2024-03-29T11:27:03.191 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.191+0000 7f927f7fe640 10 client.4515.objecter ms_handle_connect 0x7f928406dee0 2024-03-29T11:27:03.191 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.198+0000 7f927f7fe640 1 -- 172.21.15.104:0/2838881127 <== mon.0 v1:172.21.15.104:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f92880503f0 con 0x7f92980a9750 2024-03-29T11:27:03.191 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.198+0000 7f927f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:03.191 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.198+0000 7f927f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:03.510 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.520+0000 7f929d73b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:03.510 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.520+0000 7f929d73b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:03.510 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.520+0000 7f929d73b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:27:03.510 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.520+0000 7f929d73b640 1 -- 172.21.15.104:0/2838881127 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9298147ac0 con 0x7f92980a9750 2024-03-29T11:27:03.510 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.521+0000 7f927f7fe640 1 -- 172.21.15.104:0/2838881127 <== mon.0 v1:172.21.15.104:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f92880503f0 con 0x7f92980a9750 2024-03-29T11:27:03.510 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.521+0000 7f927f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:03.510 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.521+0000 7f927f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:03.510 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:27:03.511 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:03.514 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.526+0000 7f929d73b640 10 librados: watch_flush enter 2024-03-29T11:27:03.514 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.526+0000 7f929d73b640 10 librados: watch_flush exit 2024-03-29T11:27:03.514 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.526+0000 7f929d73b640 20 client.4515.objecter shutdown clearing up homeless session... 2024-03-29T11:27:03.514 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.526+0000 7f929d73b640 10 client.4515.objecter successfully canceled tick 2024-03-29T11:27:03.514 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.526+0000 7f929d73b640 1 -- 172.21.15.104:0/2838881127 >> v1:172.21.15.104:6816/3341126660 conn(0x7f928406dee0 legacy=0x7f92840703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:03.515 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.526+0000 7f929d73b640 10 monclient: shutdown 2024-03-29T11:27:03.515 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.526+0000 7f929d73b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:03.515 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.526+0000 7f929d73b640 1 -- 172.21.15.104:0/2838881127 >> v1:172.21.15.104:6789/0 conn(0x7f92980a9750 legacy=0x7f92980ac7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:03.515 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.526+0000 7f929d73b640 1 -- 172.21.15.104:0/2838881127 shutdown_connections 2024-03-29T11:27:03.515 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.526+0000 7f929d73b640 1 -- 172.21.15.104:0/2838881127 >> 172.21.15.104:0/2838881127 conn(0x7f92980b35e0 msgr2=0x7f92980b21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:03.515 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.527+0000 7f929d73b640 1 -- 172.21.15.104:0/2838881127 shutdown_connections 2024-03-29T11:27:03.515 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.527+0000 7f929d73b640 1 -- 172.21.15.104:0/2838881127 wait complete. 2024-03-29T11:27:03.515 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:03.527+0000 7f929d73b640 1 librados: shutdown 2024-03-29T11:27:03.532 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:10.340 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:10.488 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.500+0000 7f3153fff640 10 monclient: get_monmap_and_config 2024-03-29T11:27:10.488 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.500+0000 7f3153fff640 10 monclient: build_initial_monmap 2024-03-29T11:27:10.488 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.500+0000 7f3153fff640 10 monclient: monmap: 2024-03-29T11:27:10.488 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:27:10.488 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:27:10.488 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:27:10.501690+0000 2024-03-29T11:27:10.489 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:27:10.501690+0000 2024-03-29T11:27:10.489 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:10.489 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:10.489 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:27:10.489 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:27:10.489 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:27:10.489 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:10.501 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.501+0000 7f3153fff640 1 Processor -- start 2024-03-29T11:27:10.502 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.501+0000 7f3153fff640 1 -- start start 2024-03-29T11:27:10.502 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.501+0000 7f3153fff640 10 monclient: init 2024-03-29T11:27:10.502 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.502+0000 7f3153fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:10.502 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.502+0000 7f3153fff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:27:10.502 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.502+0000 7f3153fff640 10 monclient(hunting): picked mon.noname-b con 0x7f31540b4820 addr v1:172.21.15.104:6790/0 2024-03-29T11:27:10.502 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.502+0000 7f3153fff640 10 monclient(hunting): picked mon.noname-a con 0x7f31540a9ce0 addr v1:172.21.15.104:6789/0 2024-03-29T11:27:10.502 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.502+0000 7f3153fff640 10 monclient(hunting): picked mon.noname-c con 0x7f31540b3cd0 addr v1:172.21.15.119:6789/0 2024-03-29T11:27:10.502 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.502+0000 7f3153fff640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31540b4180 con 0x7f31540a9ce0 2024-03-29T11:27:10.502 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.502+0000 7f3153fff640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31540aaf60 con 0x7f31540b3cd0 2024-03-29T11:27:10.502 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.502+0000 7f3153fff640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f315413c200 con 0x7f31540b4820 2024-03-29T11:27:10.502 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.502+0000 7f3153fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:10.502 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.502+0000 7f3153fff640 10 monclient(hunting): authenticate will time out at 1411.753700s 2024-03-29T11:27:10.502 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.502+0000 7f3152ffd640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f31540b4820 0x7f315413baf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:50078/0 (socket says 172.21.15.104:50078) 2024-03-29T11:27:10.502 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.502+0000 7f3152ffd640 1 -- 172.21.15.104:0/2414442401 learned_addr learned my addr 172.21.15.104:0/2414442401 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:27:10.503 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.503+0000 7f3151ffb640 1 -- 172.21.15.104:0/2414442401 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2001750744 0 0) 0x7f3148001040 con 0x7f31540a9ce0 2024-03-29T11:27:10.503 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.503+0000 7f3151ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:10.503 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.503+0000 7f3151ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:10.503 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.503+0000 7f3151ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:10.503 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.503+0000 7f3151ffb640 10 monclient(hunting): my global_id is 4518 2024-03-29T11:27:10.503 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.503+0000 7f3151ffb640 1 -- 172.21.15.104:0/2414442401 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f313c003520 con 0x7f31540a9ce0 2024-03-29T11:27:10.503 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.503+0000 7f3151ffb640 1 -- 172.21.15.104:0/2414442401 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4206189985 0 0) 0x7f315413c200 con 0x7f31540b4820 2024-03-29T11:27:10.503 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.503+0000 7f3151ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:10.503 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.503+0000 7f3151ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:10.503 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.503+0000 7f3151ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:10.503 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.503+0000 7f3151ffb640 10 monclient(hunting): my global_id is 4532 2024-03-29T11:27:10.503 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.503+0000 7f3151ffb640 1 -- 172.21.15.104:0/2414442401 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3148001040 con 0x7f31540b4820 2024-03-29T11:27:10.503 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.503+0000 7f3151ffb640 1 -- 172.21.15.104:0/2414442401 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4156245879 0 0) 0x7f313c003520 con 0x7f31540a9ce0 2024-03-29T11:27:10.503 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.503+0000 7f3151ffb640 1 -- 172.21.15.104:0/2414442401 >> v1:172.21.15.104:6790/0 conn(0x7f31540b4820 legacy=0x7f315413baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:10.504 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.503+0000 7f3151ffb640 1 -- 172.21.15.104:0/2414442401 >> v1:172.21.15.119:6789/0 conn(0x7f31540b3cd0 legacy=0x7f315414d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:10.504 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.503+0000 7f3151ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:27:10.504 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.503+0000 7f3151ffb640 1 monclient: found mon.noname-a 2024-03-29T11:27:10.504 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.503+0000 7f3151ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:10.504 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.503+0000 7f3151ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.104:6789/0 2024-03-29T11:27:10.504 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.503+0000 7f3151ffb640 1 -- 172.21.15.104:0/2414442401 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3154145e80 con 0x7f31540a9ce0 2024-03-29T11:27:10.504 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.503+0000 7f3151ffb640 10 monclient: _finish_auth 0 2024-03-29T11:27:10.504 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.503+0000 7f3151ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:27:10.504 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.503+0000 7f3151ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:10.504 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.503+0000 7f3151ffb640 1 -- 172.21.15.104:0/2414442401 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f31480015b0 con 0x7f31540a9ce0 2024-03-29T11:27:10.504 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.503+0000 7f3151ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:10.504 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.503+0000 7f3151ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:10.504 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.503+0000 7f3151ffb640 10 monclient: dump: 2024-03-29T11:27:10.504 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:27:10.504 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:27:10.505 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:10.505 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:10.505 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:10.505 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:10.505 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:27:10.505 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:27:10.505 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:27:10.505 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:10.505 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.503+0000 7f3153fff640 5 monclient: authenticate success, global_id 4518 2024-03-29T11:27:10.505 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.503+0000 7f3153fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:10.505 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.504+0000 7f3151ffb640 1 -- 172.21.15.104:0/2414442401 <== mon.0 v1:172.21.15.104:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3148001710 con 0x7f31540a9ce0 2024-03-29T11:27:10.505 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.504+0000 7f3151ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:10.505 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.504+0000 7f3151ffb640 1 -- 172.21.15.104:0/2414442401 <== mon.0 v1:172.21.15.104:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f3148001b20 con 0x7f31540a9ce0 2024-03-29T11:27:10.505 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.504+0000 7f3151ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:10.505 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.504+0000 7f3151ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:10.506 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.504+0000 7f3151ffb640 10 monclient: dump: 2024-03-29T11:27:10.506 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:27:10.506 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:27:10.506 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:10.506 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:10.506 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:10.506 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:10.506 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:27:10.506 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:27:10.506 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:27:10.506 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:10.506 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.504+0000 7f3153fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:10.506 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.504+0000 7f3153fff640 4 set_mon_vals no callback set 2024-03-29T11:27:10.506 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.505+0000 7f3153fff640 10 monclient: shutdown 2024-03-29T11:27:10.507 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.505+0000 7f3153fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:10.507 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.505+0000 7f3153fff640 1 -- 172.21.15.104:0/2414442401 >> v1:172.21.15.104:6789/0 conn(0x7f31540a9ce0 legacy=0x7f3154145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:10.507 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.505+0000 7f3153fff640 1 -- 172.21.15.104:0/2414442401 shutdown_connections 2024-03-29T11:27:10.507 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.505+0000 7f3153fff640 1 -- 172.21.15.104:0/2414442401 >> 172.21.15.104:0/2414442401 conn(0x7f315406d070 msgr2=0x7f3154066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:10.507 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.505+0000 7f3153fff640 1 -- 172.21.15.104:0/2414442401 shutdown_connections 2024-03-29T11:27:10.507 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.505+0000 7f3153fff640 1 -- 172.21.15.104:0/2414442401 wait complete. 2024-03-29T11:27:10.507 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.506+0000 7f3153fff640 10 monclient: build_initial_monmap 2024-03-29T11:27:10.507 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.506+0000 7f3153fff640 10 monclient: monmap: 2024-03-29T11:27:10.507 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:27:10.508 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:10.508 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:27:10.508 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:27:10.508 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:10.508 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:10.508 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:27:10.508 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:27:10.508 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:27:10.508 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:10.508 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.506+0000 7f3153fff640 1 librados: starting msgr at 2024-03-29T11:27:10.508 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.506+0000 7f3153fff640 1 librados: starting objecter 2024-03-29T11:27:10.509 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.507+0000 7f3153fff640 1 Processor -- start 2024-03-29T11:27:10.509 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.507+0000 7f3153fff640 1 -- start start 2024-03-29T11:27:10.509 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.507+0000 7f3153fff640 1 librados: setting wanted keys 2024-03-29T11:27:10.509 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.507+0000 7f3153fff640 1 librados: calling monclient init 2024-03-29T11:27:10.509 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.507+0000 7f3153fff640 10 monclient: init 2024-03-29T11:27:10.509 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.507+0000 7f3153fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:10.509 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.508+0000 7f3153fff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:27:10.509 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.508+0000 7f3153fff640 10 monclient(hunting): picked mon.noname-a con 0x7f31540a9ce0 addr v1:172.21.15.104:6789/0 2024-03-29T11:27:10.509 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.508+0000 7f3153fff640 10 monclient(hunting): picked mon.noname-c con 0x7f31540b3cd0 addr v1:172.21.15.104:6790/0 2024-03-29T11:27:10.509 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.508+0000 7f3153fff640 10 monclient(hunting): picked mon.noname-b con 0x7f31540b4820 addr v1:172.21.15.119:6789/0 2024-03-29T11:27:10.509 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.508+0000 7f3152ffd640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f31540a9ce0 0x7f315406fb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:40122/0 (socket says 172.21.15.104:40122) 2024-03-29T11:27:10.509 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.508+0000 7f3152ffd640 1 -- 172.21.15.104:0/1611643767 learned_addr learned my addr 172.21.15.104:0/1611643767 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:27:10.509 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.508+0000 7f3153fff640 1 -- 172.21.15.104:0/1611643767 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31541dfda0 con 0x7f31540a9ce0 2024-03-29T11:27:10.509 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.508+0000 7f313b7fe640 10 client.?.objecter ms_handle_connect 0x7f31540a9ce0 2024-03-29T11:27:10.510 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.508+0000 7f313b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:10.510 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.508+0000 7f313b7fe640 10 client.?.objecter ms_handle_connect 0x7f31540b3cd0 2024-03-29T11:27:10.510 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.508+0000 7f313b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:10.510 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.508+0000 7f3153fff640 1 -- 172.21.15.104:0/1611643767 --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f315406f410 con 0x7f31540b4820 2024-03-29T11:27:10.510 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.508+0000 7f3153fff640 1 -- 172.21.15.104:0/1611643767 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31541335a0 con 0x7f31540b3cd0 2024-03-29T11:27:10.510 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.508+0000 7f3153fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:10.510 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.508+0000 7f3153fff640 10 monclient(hunting): authenticate will time out at 1411.759598s 2024-03-29T11:27:10.510 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.508+0000 7f313b7fe640 1 -- 172.21.15.104:0/1611643767 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3419716869 0 0) 0x7f31541dfda0 con 0x7f31540a9ce0 2024-03-29T11:27:10.510 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.508+0000 7f313b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:10.510 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.508+0000 7f313b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:10.510 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.508+0000 7f313b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:10.510 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.508+0000 7f313b7fe640 10 monclient(hunting): my global_id is 4521 2024-03-29T11:27:10.510 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.508+0000 7f313b7fe640 1 -- 172.21.15.104:0/1611643767 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3140004eb0 con 0x7f31540a9ce0 2024-03-29T11:27:10.510 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.509+0000 7f313b7fe640 1 -- 172.21.15.104:0/1611643767 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2915908383 0 0) 0x7f3140004eb0 con 0x7f31540a9ce0 2024-03-29T11:27:10.510 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.509+0000 7f313b7fe640 1 -- 172.21.15.104:0/1611643767 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f31541dfda0 con 0x7f31540a9ce0 2024-03-29T11:27:10.511 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.509+0000 7f313b7fe640 1 -- 172.21.15.104:0/1611643767 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f314c001c90 con 0x7f31540a9ce0 2024-03-29T11:27:10.511 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.509+0000 7f313b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:10.511 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.509+0000 7f313b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:10.511 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.509+0000 7f313b7fe640 10 monclient(hunting): dump: 2024-03-29T11:27:10.511 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:27:10.511 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:27:10.511 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:10.511 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:10.511 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:10.511 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:10.511 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:27:10.511 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:27:10.511 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:27:10.511 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:10.511 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.509+0000 7f313b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:10.512 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.509+0000 7f313b7fe640 1 -- 172.21.15.104:0/1611643767 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3283074915 0 0) 0x7f31541335a0 con 0x7f31540b3cd0 2024-03-29T11:27:10.512 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.509+0000 7f313b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:10.512 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.509+0000 7f313b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:10.512 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.509+0000 7f313b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:10.512 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.509+0000 7f313b7fe640 10 monclient(hunting): my global_id is 4535 2024-03-29T11:27:10.512 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.509+0000 7f313b7fe640 1 -- 172.21.15.104:0/1611643767 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3140004eb0 con 0x7f31540b3cd0 2024-03-29T11:27:10.512 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.509+0000 7f313b7fe640 1 -- 172.21.15.104:0/1611643767 <== mon.0 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 864806067 0 0) 0x7f31541dfda0 con 0x7f31540a9ce0 2024-03-29T11:27:10.512 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.509+0000 7f313b7fe640 1 -- 172.21.15.104:0/1611643767 >> v1:172.21.15.104:6790/0 conn(0x7f31540b3cd0 legacy=0x7f31541def80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:10.512 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.509+0000 7f313b7fe640 1 -- 172.21.15.104:0/1611643767 >> v1:172.21.15.119:6789/0 conn(0x7f31540b4820 legacy=0x7f31541df690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:10.512 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.509+0000 7f313b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:27:10.512 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.509+0000 7f313b7fe640 1 monclient: found mon.a 2024-03-29T11:27:10.512 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.509+0000 7f313b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:10.512 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.509+0000 7f313b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:27:10.512 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.509+0000 7f313b7fe640 1 -- 172.21.15.104:0/1611643767 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f315406f130 con 0x7f31540a9ce0 2024-03-29T11:27:10.512 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.509+0000 7f313b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:27:10.513 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.509+0000 7f313b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:27:10.513 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.509+0000 7f313b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:10.513 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.509+0000 7f3153fff640 5 monclient: authenticate success, global_id 4521 2024-03-29T11:27:10.513 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.509+0000 7f3153fff640 10 monclient: _renew_subs 2024-03-29T11:27:10.513 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.509+0000 7f3153fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:27:10.513 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.509+0000 7f3153fff640 1 -- 172.21.15.104:0/1611643767 --> v1:172.21.15.104:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3154070220 con 0x7f31540a9ce0 2024-03-29T11:27:10.513 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.509+0000 7f3153fff640 10 client.4521.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:10.513 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.509+0000 7f3153fff640 10 monclient: _renew_subs 2024-03-29T11:27:10.513 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.509+0000 7f3153fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:27:10.513 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.509+0000 7f3153fff640 1 -- 172.21.15.104:0/1611643767 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3154145240 con 0x7f31540a9ce0 2024-03-29T11:27:10.513 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.509+0000 7f3153fff640 1 librados: init done 2024-03-29T11:27:10.514 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.510+0000 7f313b7fe640 1 -- 172.21.15.104:0/1611643767 <== mon.0 v1:172.21.15.104:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f314c004b60 con 0x7f31540a9ce0 2024-03-29T11:27:10.514 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.510+0000 7f313b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:10.514 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.510+0000 7f313b7fe640 1 -- 172.21.15.104:0/1611643767 <== mon.0 v1:172.21.15.104:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f314c004e70 con 0x7f31540a9ce0 2024-03-29T11:27:10.514 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.510+0000 7f313b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:10.514 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.510+0000 7f313b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:10.514 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.510+0000 7f313b7fe640 10 monclient: dump: 2024-03-29T11:27:10.514 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:27:10.514 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:27:10.514 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:10.514 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:10.514 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:10.514 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:10.514 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:27:10.515 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:27:10.515 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:27:10.515 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:10.515 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.510+0000 7f31517fa640 4 set_mon_vals no callback set 2024-03-29T11:27:10.515 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.511+0000 7f3153fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:10.515 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.511+0000 7f3153fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:10.515 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.511+0000 7f3153fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:27:10.515 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.511+0000 7f3153fff640 1 -- 172.21.15.104:0/1611643767 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3134005080 con 0x7f31540a9ce0 2024-03-29T11:27:10.515 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.513+0000 7f313b7fe640 1 -- 172.21.15.104:0/1611643767 <== mon.0 v1:172.21.15.104:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f314c01ae40 con 0x7f31540a9ce0 2024-03-29T11:27:10.516 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.514+0000 7f31527fc640 10 monclient: get_auth_request con 0x7f314006dd30 auth_method 0 2024-03-29T11:27:10.516 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.514+0000 7f313b7fe640 1 -- 172.21.15.104:0/1611643767 <== mon.0 v1:172.21.15.104:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f314c054b80 con 0x7f31540a9ce0 2024-03-29T11:27:10.516 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.514+0000 7f313b7fe640 10 client.4521.objecter ms_dispatch 0x7f3154000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:10.516 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.514+0000 7f313b7fe640 3 client.4521.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:10.517 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.514+0000 7f313b7fe640 3 client.4521.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:10.517 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.515+0000 7f313b7fe640 20 client.4521.objecter dump_active .. 0 homeless 2024-03-29T11:27:10.517 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.515+0000 7f313b7fe640 10 client.4521.objecter ms_handle_connect 0x7f314006dd30 2024-03-29T11:27:10.517 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.520+0000 7f313b7fe640 1 -- 172.21.15.104:0/1611643767 <== mon.0 v1:172.21.15.104:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f314c0504c0 con 0x7f31540a9ce0 2024-03-29T11:27:10.517 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.520+0000 7f313b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:10.517 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.520+0000 7f313b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:10.866 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.878+0000 7f3153fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:10.866 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.878+0000 7f3153fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:10.866 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.878+0000 7f3153fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:27:10.866 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.878+0000 7f3153fff640 1 -- 172.21.15.104:0/1611643767 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3134005330 con 0x7f31540a9ce0 2024-03-29T11:27:10.866 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.879+0000 7f313b7fe640 1 -- 172.21.15.104:0/1611643767 <== mon.0 v1:172.21.15.104:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f314c0525c0 con 0x7f31540a9ce0 2024-03-29T11:27:10.867 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.879+0000 7f313b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:10.867 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.879+0000 7f313b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:10.867 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:27:10.867 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:10.870 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.882+0000 7f3153fff640 10 librados: watch_flush enter 2024-03-29T11:27:10.870 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.882+0000 7f3153fff640 10 librados: watch_flush exit 2024-03-29T11:27:10.871 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.882+0000 7f3153fff640 20 client.4521.objecter shutdown clearing up homeless session... 2024-03-29T11:27:10.871 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.882+0000 7f3153fff640 10 client.4521.objecter successfully canceled tick 2024-03-29T11:27:10.871 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.882+0000 7f3153fff640 1 -- 172.21.15.104:0/1611643767 >> v1:172.21.15.104:6816/3341126660 conn(0x7f314006dd30 legacy=0x7f31400701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:10.871 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.882+0000 7f3153fff640 10 monclient: shutdown 2024-03-29T11:27:10.871 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.882+0000 7f3153fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:10.871 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.882+0000 7f3153fff640 1 -- 172.21.15.104:0/1611643767 >> v1:172.21.15.104:6789/0 conn(0x7f31540a9ce0 legacy=0x7f315406fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:10.871 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.883+0000 7f3153fff640 1 -- 172.21.15.104:0/1611643767 shutdown_connections 2024-03-29T11:27:10.871 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.883+0000 7f3153fff640 1 -- 172.21.15.104:0/1611643767 >> 172.21.15.104:0/1611643767 conn(0x7f315406d070 msgr2=0x7f3154141f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:10.871 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.883+0000 7f3153fff640 1 -- 172.21.15.104:0/1611643767 shutdown_connections 2024-03-29T11:27:10.871 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.883+0000 7f3153fff640 1 -- 172.21.15.104:0/1611643767 wait complete. 2024-03-29T11:27:10.871 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:10.883+0000 7f3153fff640 1 librados: shutdown 2024-03-29T11:27:10.886 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:17.694 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:17.849 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.860+0000 7f50f2a74640 10 monclient: get_monmap_and_config 2024-03-29T11:27:17.849 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.860+0000 7f50f2a74640 10 monclient: build_initial_monmap 2024-03-29T11:27:17.850 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.860+0000 7f50f2a74640 10 monclient: monmap: 2024-03-29T11:27:17.850 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:27:17.850 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:27:17.850 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:27:17.861820+0000 2024-03-29T11:27:17.850 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:27:17.861820+0000 2024-03-29T11:27:17.850 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:17.850 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:17.850 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:27:17.850 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:27:17.850 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:27:17.850 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:17.851 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.863+0000 7f50f2a74640 1 Processor -- start 2024-03-29T11:27:17.856 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.863+0000 7f50f2a74640 1 -- start start 2024-03-29T11:27:17.856 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.863+0000 7f50f2a74640 10 monclient: init 2024-03-29T11:27:17.856 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.864+0000 7f50f2a74640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:17.856 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.864+0000 7f50f2a74640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:27:17.856 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.864+0000 7f50f2a74640 10 monclient(hunting): picked mon.noname-b con 0x7f50ec0a9ac0 addr v1:172.21.15.104:6790/0 2024-03-29T11:27:17.856 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.864+0000 7f50f2a74640 10 monclient(hunting): picked mon.noname-a con 0x7f50ec13ee90 addr v1:172.21.15.104:6789/0 2024-03-29T11:27:17.856 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.864+0000 7f50f2a74640 10 monclient(hunting): picked mon.noname-c con 0x7f50ec0af900 addr v1:172.21.15.119:6789/0 2024-03-29T11:27:17.856 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.864+0000 7f50f2a74640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50ec0afd10 con 0x7f50ec13ee90 2024-03-29T11:27:17.856 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.864+0000 7f50f2a74640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50ec0af6f0 con 0x7f50ec0af900 2024-03-29T11:27:17.856 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.864+0000 7f50f2a74640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50ec0aac80 con 0x7f50ec0a9ac0 2024-03-29T11:27:17.856 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.864+0000 7f50f2a74640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:17.856 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.864+0000 7f50f2a74640 10 monclient(hunting): authenticate will time out at 1419.115709s 2024-03-29T11:27:17.857 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.868+0000 7f50f0810640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f50ec0a9ac0 0x7f50ec0adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:37565/0 (socket says 172.21.15.104:37565) 2024-03-29T11:27:17.857 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.868+0000 7f50f0810640 1 -- 172.21.15.104:0/3571567700 learned_addr learned my addr 172.21.15.104:0/3571567700 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:27:17.861 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.872+0000 7f50eb7fe640 1 -- 172.21.15.104:0/3571567700 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 213800292 0 0) 0x7f50ec0af6f0 con 0x7f50ec0af900 2024-03-29T11:27:17.861 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.872+0000 7f50eb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:17.861 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.872+0000 7f50eb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:17.861 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.872+0000 7f50eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:17.861 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.872+0000 7f50eb7fe640 10 monclient(hunting): my global_id is 4498 2024-03-29T11:27:17.861 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.872+0000 7f50eb7fe640 1 -- 172.21.15.104:0/3571567700 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50d4003520 con 0x7f50ec0af900 2024-03-29T11:27:17.861 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.872+0000 7f50eb7fe640 1 -- 172.21.15.104:0/3571567700 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3305716300 0 0) 0x7f50ec0aac80 con 0x7f50ec0a9ac0 2024-03-29T11:27:17.861 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.872+0000 7f50eb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:17.861 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.872+0000 7f50eb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:17.861 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.872+0000 7f50eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:17.862 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.872+0000 7f50eb7fe640 10 monclient(hunting): my global_id is 4538 2024-03-29T11:27:17.862 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.872+0000 7f50eb7fe640 1 -- 172.21.15.104:0/3571567700 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50ec0af6f0 con 0x7f50ec0a9ac0 2024-03-29T11:27:17.862 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.872+0000 7f50eb7fe640 1 -- 172.21.15.104:0/3571567700 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 39226383 0 0) 0x7f50ec0afd10 con 0x7f50ec13ee90 2024-03-29T11:27:17.862 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.872+0000 7f50eb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:17.862 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.872+0000 7f50eb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:17.862 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.872+0000 7f50eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:17.862 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.872+0000 7f50eb7fe640 10 monclient(hunting): my global_id is 4524 2024-03-29T11:27:17.862 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.873+0000 7f50eb7fe640 1 -- 172.21.15.104:0/3571567700 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50ec0aac80 con 0x7f50ec13ee90 2024-03-29T11:27:17.862 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.873+0000 7f50eb7fe640 1 -- 172.21.15.104:0/3571567700 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2373351660 0 0) 0x7f50ec0af6f0 con 0x7f50ec0a9ac0 2024-03-29T11:27:17.862 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.873+0000 7f50eb7fe640 1 -- 172.21.15.104:0/3571567700 >> v1:172.21.15.119:6789/0 conn(0x7f50ec0af900 legacy=0x7f50ec149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:17.862 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.873+0000 7f50eb7fe640 1 -- 172.21.15.104:0/3571567700 >> v1:172.21.15.104:6789/0 conn(0x7f50ec13ee90 legacy=0x7f50ec13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:17.862 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.873+0000 7f50eb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:27:17.862 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.873+0000 7f50eb7fe640 1 monclient: found mon.noname-b 2024-03-29T11:27:17.862 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.873+0000 7f50eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:17.863 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.873+0000 7f50eb7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.104:6790/0 2024-03-29T11:27:17.863 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.873+0000 7f50eb7fe640 1 -- 172.21.15.104:0/3571567700 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50ec0aaf60 con 0x7f50ec0a9ac0 2024-03-29T11:27:17.863 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.873+0000 7f50eb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:27:17.863 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.873+0000 7f50eb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:27:17.863 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.873+0000 7f50eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:17.863 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.873+0000 7f50eb7fe640 1 -- 172.21.15.104:0/3571567700 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f50e0001520 con 0x7f50ec0a9ac0 2024-03-29T11:27:17.863 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.873+0000 7f50eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:17.863 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.873+0000 7f50eb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:17.863 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.873+0000 7f50eb7fe640 10 monclient: dump: 2024-03-29T11:27:17.863 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:27:17.863 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:27:17.863 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:17.863 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:17.864 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:17.864 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:17.864 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:27:17.864 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:27:17.864 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:27:17.865 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:17.865 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.873+0000 7f50f2a74640 5 monclient: authenticate success, global_id 4538 2024-03-29T11:27:17.865 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.873+0000 7f50f2a74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:17.865 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.874+0000 7f50eb7fe640 1 -- 172.21.15.104:0/3571567700 <== mon.2 v1:172.21.15.104:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f50e0001680 con 0x7f50ec0a9ac0 2024-03-29T11:27:17.866 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.874+0000 7f50eb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:17.866 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.874+0000 7f50eb7fe640 1 -- 172.21.15.104:0/3571567700 <== mon.2 v1:172.21.15.104:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f50e0001a90 con 0x7f50ec0a9ac0 2024-03-29T11:27:17.866 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.874+0000 7f50eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:17.866 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.874+0000 7f50eb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:17.866 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.874+0000 7f50eb7fe640 10 monclient: dump: 2024-03-29T11:27:17.866 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:27:17.866 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:27:17.866 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:17.866 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:17.866 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:17.866 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:17.866 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:27:17.866 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:27:17.867 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:27:17.867 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:17.867 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.874+0000 7f50f2a74640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:17.867 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.874+0000 7f50f2a74640 4 set_mon_vals no callback set 2024-03-29T11:27:17.867 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.875+0000 7f50f2a74640 10 monclient: shutdown 2024-03-29T11:27:17.867 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.875+0000 7f50f2a74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:17.867 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.875+0000 7f50f2a74640 1 -- 172.21.15.104:0/3571567700 >> v1:172.21.15.104:6790/0 conn(0x7f50ec0a9ac0 legacy=0x7f50ec0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:17.869 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.878+0000 7f50f2a74640 1 -- 172.21.15.104:0/3571567700 shutdown_connections 2024-03-29T11:27:17.869 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.878+0000 7f50f2a74640 1 -- 172.21.15.104:0/3571567700 >> 172.21.15.104:0/3571567700 conn(0x7f50ec0b35e0 msgr2=0x7f50ec0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:17.869 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.878+0000 7f50f2a74640 1 -- 172.21.15.104:0/3571567700 shutdown_connections 2024-03-29T11:27:17.869 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.879+0000 7f50f2a74640 1 -- 172.21.15.104:0/3571567700 wait complete. 2024-03-29T11:27:17.869 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.879+0000 7f50f2a74640 10 monclient: build_initial_monmap 2024-03-29T11:27:17.869 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.879+0000 7f50f2a74640 10 monclient: monmap: 2024-03-29T11:27:17.869 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:27:17.869 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:17.869 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:27:17.869 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:27:17.869 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:17.869 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:17.870 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:27:17.870 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:27:17.870 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:27:17.870 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:17.870 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.880+0000 7f50f2a74640 1 librados: starting msgr at 2024-03-29T11:27:17.870 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.880+0000 7f50f2a74640 1 librados: starting objecter 2024-03-29T11:27:17.870 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.880+0000 7f50f2a74640 1 Processor -- start 2024-03-29T11:27:17.882 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.881+0000 7f50f2a74640 1 -- start start 2024-03-29T11:27:17.882 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.881+0000 7f50f2a74640 1 librados: setting wanted keys 2024-03-29T11:27:17.882 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.881+0000 7f50f2a74640 1 librados: calling monclient init 2024-03-29T11:27:17.882 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.881+0000 7f50f2a74640 10 monclient: init 2024-03-29T11:27:17.882 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.882+0000 7f50f2a74640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:17.882 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.882+0000 7f50f2a74640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:27:17.882 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.882+0000 7f50f2a74640 10 monclient(hunting): picked mon.noname-b con 0x7f50ec0a9ac0 addr v1:172.21.15.119:6789/0 2024-03-29T11:27:17.882 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.882+0000 7f50f2a74640 10 monclient(hunting): picked mon.noname-c con 0x7f50ec0af900 addr v1:172.21.15.104:6790/0 2024-03-29T11:27:17.882 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.882+0000 7f50f2a74640 10 monclient(hunting): picked mon.noname-a con 0x7f50ec13ee90 addr v1:172.21.15.104:6789/0 2024-03-29T11:27:17.882 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.882+0000 7f50f2a74640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50ec1e41e0 con 0x7f50ec13ee90 2024-03-29T11:27:17.883 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.882+0000 7f50f2a74640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50ec1e5060 con 0x7f50ec0a9ac0 2024-03-29T11:27:17.883 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.882+0000 7f50f2a74640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50ec1c66a0 con 0x7f50ec0af900 2024-03-29T11:27:17.883 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.882+0000 7f50f2a74640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:17.883 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.882+0000 7f50f2a74640 10 monclient(hunting): authenticate will time out at 1419.133339s 2024-03-29T11:27:17.883 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.885+0000 7f50f0810640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f50ec0a9ac0 0x7f50ec1c4640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:47977/0 (socket says 172.21.15.104:47977) 2024-03-29T11:27:17.883 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.885+0000 7f50f0810640 1 -- 172.21.15.104:0/3077468210 learned_addr learned my addr 172.21.15.104:0/3077468210 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:27:17.883 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.886+0000 7f50e97fa640 10 client.?.objecter ms_handle_connect 0x7f50ec0a9ac0 2024-03-29T11:27:17.883 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.886+0000 7f50e97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:17.883 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.886+0000 7f50e97fa640 1 -- 172.21.15.104:0/3077468210 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3106160912 0 0) 0x7f50ec1e5060 con 0x7f50ec0a9ac0 2024-03-29T11:27:17.883 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.886+0000 7f50e97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:27:17.883 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.886+0000 7f50e97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:17.883 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.886+0000 7f50e97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:17.884 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.886+0000 7f50e97fa640 10 monclient(hunting): my global_id is 4501 2024-03-29T11:27:17.884 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.886+0000 7f50e97fa640 1 -- 172.21.15.104:0/3077468210 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50d8004eb0 con 0x7f50ec0a9ac0 2024-03-29T11:27:17.884 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.886+0000 7f50e97fa640 1 -- 172.21.15.104:0/3077468210 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1415589738 0 0) 0x7f50d8004eb0 con 0x7f50ec0a9ac0 2024-03-29T11:27:17.884 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.887+0000 7f50e97fa640 1 -- 172.21.15.104:0/3077468210 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50ec1e5060 con 0x7f50ec0a9ac0 2024-03-29T11:27:17.884 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.887+0000 7f50e97fa640 1 -- 172.21.15.104:0/3077468210 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f50e00019d0 con 0x7f50ec0a9ac0 2024-03-29T11:27:17.884 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.887+0000 7f50e97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:17.884 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.887+0000 7f50e97fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:17.884 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.887+0000 7f50e97fa640 10 monclient(hunting): dump: 2024-03-29T11:27:17.884 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:27:17.884 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:27:17.884 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:17.884 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:17.884 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:17.884 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:17.885 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:27:17.885 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:27:17.885 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:27:17.885 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:17.885 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.887+0000 7f50e97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:17.885 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.888+0000 7f50e97fa640 1 -- 172.21.15.104:0/3077468210 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 791082368 0 0) 0x7f50ec1e5060 con 0x7f50ec0a9ac0 2024-03-29T11:27:17.885 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.888+0000 7f50e97fa640 1 -- 172.21.15.104:0/3077468210 >> v1:172.21.15.104:6790/0 conn(0x7f50ec0af900 legacy=0x7f50ec1e4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:17.885 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.888+0000 7f50e97fa640 1 -- 172.21.15.104:0/3077468210 >> v1:172.21.15.104:6789/0 conn(0x7f50ec13ee90 legacy=0x7f50ec1e1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:17.885 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.888+0000 7f50e97fa640 10 monclient: _finish_hunting 0 2024-03-29T11:27:17.886 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.888+0000 7f50e97fa640 1 monclient: found mon.b 2024-03-29T11:27:17.886 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.888+0000 7f50e97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:17.886 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.888+0000 7f50e97fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:27:17.886 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.888+0000 7f50e97fa640 1 -- 172.21.15.104:0/3077468210 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50ec1e4430 con 0x7f50ec0a9ac0 2024-03-29T11:27:17.886 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.888+0000 7f50e97fa640 10 monclient: _finish_auth 0 2024-03-29T11:27:17.886 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.888+0000 7f50e97fa640 10 monclient: _check_auth_tickets 2024-03-29T11:27:17.886 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.888+0000 7f50e97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:17.886 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.888+0000 7f50f2a74640 5 monclient: authenticate success, global_id 4501 2024-03-29T11:27:17.886 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.888+0000 7f50f2a74640 10 monclient: _renew_subs 2024-03-29T11:27:17.886 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.888+0000 7f50f2a74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:27:17.887 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.888+0000 7f50f2a74640 1 -- 172.21.15.104:0/3077468210 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50ec06ffc0 con 0x7f50ec0a9ac0 2024-03-29T11:27:17.887 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.888+0000 7f50f2a74640 10 client.4501.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:17.887 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.888+0000 7f50f2a74640 10 monclient: _renew_subs 2024-03-29T11:27:17.887 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.888+0000 7f50f2a74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:27:17.887 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.888+0000 7f50f2a74640 1 -- 172.21.15.104:0/3077468210 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f50ec1c4e50 con 0x7f50ec0a9ac0 2024-03-29T11:27:17.887 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.888+0000 7f50f2a74640 1 librados: init done 2024-03-29T11:27:17.887 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.889+0000 7f50e97fa640 1 -- 172.21.15.104:0/3077468210 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f50e0001b30 con 0x7f50ec0a9ac0 2024-03-29T11:27:17.887 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.889+0000 7f50e97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:17.887 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.890+0000 7f50e97fa640 1 -- 172.21.15.104:0/3077468210 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f50e0004c70 con 0x7f50ec0a9ac0 2024-03-29T11:27:17.887 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.890+0000 7f50e97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:17.887 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.890+0000 7f50e97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:17.887 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.890+0000 7f50e97fa640 10 monclient: dump: 2024-03-29T11:27:17.887 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:27:17.888 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:27:17.888 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:17.888 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:17.888 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:17.888 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:17.888 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:27:17.888 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:27:17.888 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:27:17.888 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:17.888 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.891+0000 7f50eaffd640 4 set_mon_vals no callback set 2024-03-29T11:27:17.889 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.891+0000 7f50f2a74640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:17.889 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.892+0000 7f50e97fa640 1 -- 172.21.15.104:0/3077468210 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f50e0005ec0 con 0x7f50ec0a9ac0 2024-03-29T11:27:17.889 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.893+0000 7f50e97fa640 1 -- 172.21.15.104:0/3077468210 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f50e0084c60 con 0x7f50ec0a9ac0 2024-03-29T11:27:17.889 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.893+0000 7f50e97fa640 10 client.4501.objecter ms_dispatch 0x7f50ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:17.889 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.893+0000 7f50e97fa640 3 client.4501.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:17.889 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.893+0000 7f50e97fa640 3 client.4501.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:17.890 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.893+0000 7f50e97fa640 20 client.4501.objecter dump_active .. 0 homeless 2024-03-29T11:27:17.890 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.893+0000 7f50f2a74640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:17.890 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.893+0000 7f50f2a74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:27:17.890 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.893+0000 7f50f2a74640 1 -- 172.21.15.104:0/3077468210 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50cc005080 con 0x7f50ec0a9ac0 2024-03-29T11:27:17.890 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.894+0000 7f50ebfff640 10 monclient: get_auth_request con 0x7f50d806dd50 auth_method 0 2024-03-29T11:27:17.890 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.894+0000 7f50e97fa640 10 client.4501.objecter ms_handle_connect 0x7f50d806dd50 2024-03-29T11:27:17.890 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.903+0000 7f50e97fa640 1 -- 172.21.15.104:0/3077468210 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f50e0050370 con 0x7f50ec0a9ac0 2024-03-29T11:27:17.890 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.903+0000 7f50e97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:17.891 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:17.903+0000 7f50e97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:18.212 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:18.224+0000 7f50f2a74640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:18.212 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:18.224+0000 7f50f2a74640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:18.212 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:18.224+0000 7f50f2a74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:27:18.212 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:18.224+0000 7f50f2a74640 1 -- 172.21.15.104:0/3077468210 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f50cc005330 con 0x7f50ec0a9ac0 2024-03-29T11:27:18.217 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:18.230+0000 7f50e97fa640 1 -- 172.21.15.104:0/3077468210 <== mon.1 v1:172.21.15.119:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f50e0052470 con 0x7f50ec0a9ac0 2024-03-29T11:27:18.217 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:18.230+0000 7f50e97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:18.217 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:18.230+0000 7f50e97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:18.219 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:27:18.219 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:18.223 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:18.235+0000 7f50f2a74640 10 librados: watch_flush enter 2024-03-29T11:27:18.223 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:18.236+0000 7f50f2a74640 10 librados: watch_flush exit 2024-03-29T11:27:18.224 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:18.236+0000 7f50f2a74640 20 client.4501.objecter shutdown clearing up homeless session... 2024-03-29T11:27:18.224 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:18.236+0000 7f50f2a74640 10 client.4501.objecter successfully canceled tick 2024-03-29T11:27:18.224 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:18.236+0000 7f50f2a74640 1 -- 172.21.15.104:0/3077468210 >> v1:172.21.15.104:6816/3341126660 conn(0x7f50d806dd50 legacy=0x7f50d8070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:18.224 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:18.236+0000 7f50f2a74640 10 monclient: shutdown 2024-03-29T11:27:18.224 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:18.236+0000 7f50f2a74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:18.224 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:18.236+0000 7f50f2a74640 1 -- 172.21.15.104:0/3077468210 >> v1:172.21.15.119:6789/0 conn(0x7f50ec0a9ac0 legacy=0x7f50ec1c4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:18.224 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:18.236+0000 7f50f2a74640 1 -- 172.21.15.104:0/3077468210 shutdown_connections 2024-03-29T11:27:18.224 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:18.236+0000 7f50f2a74640 1 -- 172.21.15.104:0/3077468210 >> 172.21.15.104:0/3077468210 conn(0x7f50ec0b35e0 msgr2=0x7f50ec13af70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:18.225 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:18.236+0000 7f50f2a74640 1 -- 172.21.15.104:0/3077468210 shutdown_connections 2024-03-29T11:27:18.225 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:18.236+0000 7f50f2a74640 1 -- 172.21.15.104:0/3077468210 wait complete. 2024-03-29T11:27:18.225 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:18.236+0000 7f50f2a74640 1 librados: shutdown 2024-03-29T11:27:18.241 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:25.048 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:25.195 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.207+0000 7ff3a359e640 10 monclient: get_monmap_and_config 2024-03-29T11:27:25.195 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.207+0000 7ff3a359e640 10 monclient: build_initial_monmap 2024-03-29T11:27:25.195 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.207+0000 7ff3a359e640 10 monclient: monmap: 2024-03-29T11:27:25.195 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:27:25.195 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:27:25.195 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:27:25.208143+0000 2024-03-29T11:27:25.196 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:27:25.208143+0000 2024-03-29T11:27:25.196 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:25.196 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:25.196 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:27:25.196 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:27:25.196 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:27:25.196 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:25.196 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.208+0000 7ff3a359e640 1 Processor -- start 2024-03-29T11:27:25.197 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.208+0000 7ff3a359e640 1 -- start start 2024-03-29T11:27:25.197 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.208+0000 7ff3a359e640 10 monclient: init 2024-03-29T11:27:25.200 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.209+0000 7ff3a359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:25.201 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.209+0000 7ff3a359e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:27:25.201 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.209+0000 7ff3a359e640 10 monclient(hunting): picked mon.noname-c con 0x7ff3a4143a90 addr v1:172.21.15.119:6789/0 2024-03-29T11:27:25.201 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.209+0000 7ff3a359e640 10 monclient(hunting): picked mon.noname-a con 0x7ff3a4146fd0 addr v1:172.21.15.104:6789/0 2024-03-29T11:27:25.201 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.212+0000 7ff3a259c640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7ff3a4143a90 0x7ff3a4146320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:44292/0 (socket says 172.21.15.104:44292) 2024-03-29T11:27:25.201 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.212+0000 7ff3a359e640 10 monclient(hunting): picked mon.noname-b con 0x7ff3a40b2e50 addr v1:172.21.15.104:6790/0 2024-03-29T11:27:25.201 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.212+0000 7ff3a259c640 1 -- 172.21.15.104:0/977159306 learned_addr learned my addr 172.21.15.104:0/977159306 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:27:25.201 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.212+0000 7ff3a359e640 1 -- 172.21.15.104:0/977159306 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3a406ef90 con 0x7ff3a4146fd0 2024-03-29T11:27:25.201 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.212+0000 7ff3a359e640 1 -- 172.21.15.104:0/977159306 --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3a406f1c0 con 0x7ff3a4143a90 2024-03-29T11:27:25.201 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.212+0000 7ff3a359e640 1 -- 172.21.15.104:0/977159306 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3a4071eb0 con 0x7ff3a40b2e50 2024-03-29T11:27:25.201 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.212+0000 7ff3a359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:25.201 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.212+0000 7ff3a359e640 10 monclient(hunting): authenticate will time out at 1426.463875s 2024-03-29T11:27:25.202 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.213+0000 7ff3a159a640 1 -- 172.21.15.104:0/977159306 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4040629841 0 0) 0x7ff3a406ef90 con 0x7ff3a4146fd0 2024-03-29T11:27:25.202 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.213+0000 7ff3a159a640 10 monclient(hunting): _negotiate 2024-03-29T11:27:25.202 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.213+0000 7ff3a159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:25.202 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.213+0000 7ff3a159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:25.202 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.213+0000 7ff3a159a640 10 monclient(hunting): my global_id is 4527 2024-03-29T11:27:25.202 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.213+0000 7ff3a159a640 1 -- 172.21.15.104:0/977159306 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff38c003520 con 0x7ff3a4146fd0 2024-03-29T11:27:25.202 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.214+0000 7ff3a159a640 1 -- 172.21.15.104:0/977159306 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 189080154 0 0) 0x7ff38c003520 con 0x7ff3a4146fd0 2024-03-29T11:27:25.202 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.214+0000 7ff3a159a640 1 -- 172.21.15.104:0/977159306 >> v1:172.21.15.104:6790/0 conn(0x7ff3a40b2e50 legacy=0x7ff3a4151c40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:25.206 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.216+0000 7ff3a159a640 1 -- 172.21.15.104:0/977159306 >> v1:172.21.15.119:6789/0 conn(0x7ff3a4143a90 legacy=0x7ff3a4146320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:25.206 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.216+0000 7ff3a159a640 10 monclient: _finish_hunting 0 2024-03-29T11:27:25.206 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.216+0000 7ff3a159a640 1 monclient: found mon.noname-a 2024-03-29T11:27:25.206 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.216+0000 7ff3a159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:25.206 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.216+0000 7ff3a159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.104:6789/0 2024-03-29T11:27:25.206 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.216+0000 7ff3a159a640 1 -- 172.21.15.104:0/977159306 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3a40b3300 con 0x7ff3a4146fd0 2024-03-29T11:27:25.206 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.216+0000 7ff3a159a640 10 monclient: _finish_auth 0 2024-03-29T11:27:25.206 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.216+0000 7ff3a159a640 10 monclient: _check_auth_tickets 2024-03-29T11:27:25.206 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.216+0000 7ff3a159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:25.207 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.216+0000 7ff3a159a640 1 -- 172.21.15.104:0/977159306 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7ff394001690 con 0x7ff3a4146fd0 2024-03-29T11:27:25.207 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.216+0000 7ff3a159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:25.207 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.216+0000 7ff3a159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:25.207 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.217+0000 7ff3a159a640 10 monclient: dump: 2024-03-29T11:27:25.207 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:27:25.207 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:27:25.207 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:25.207 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:25.207 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:25.207 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:25.207 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:27:25.207 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:27:25.207 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:27:25.207 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:25.207 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.217+0000 7ff3a159a640 1 -- 172.21.15.104:0/977159306 <== mon.0 v1:172.21.15.104:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff394001920 con 0x7ff3a4146fd0 2024-03-29T11:27:25.208 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.217+0000 7ff3a159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:25.208 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.217+0000 7ff3a159a640 1 -- 172.21.15.104:0/977159306 <== mon.0 v1:172.21.15.104:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7ff394001cb0 con 0x7ff3a4146fd0 2024-03-29T11:27:25.208 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.217+0000 7ff3a159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:25.208 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.217+0000 7ff3a159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:25.208 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.217+0000 7ff3a159a640 10 monclient: dump: 2024-03-29T11:27:25.208 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:27:25.208 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:27:25.208 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:25.208 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:25.208 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:25.208 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:25.208 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:27:25.208 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:27:25.208 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:27:25.208 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:25.209 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.217+0000 7ff3a359e640 5 monclient: authenticate success, global_id 4527 2024-03-29T11:27:25.209 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.217+0000 7ff3a359e640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:25.212 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.217+0000 7ff3a359e640 4 set_mon_vals no callback set 2024-03-29T11:27:25.212 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.218+0000 7ff3a359e640 10 monclient: shutdown 2024-03-29T11:27:25.212 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.218+0000 7ff3a359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:25.212 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.218+0000 7ff3a359e640 1 -- 172.21.15.104:0/977159306 >> v1:172.21.15.104:6789/0 conn(0x7ff3a4146fd0 legacy=0x7ff3a40b2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:25.212 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.219+0000 7ff3a359e640 1 -- 172.21.15.104:0/977159306 shutdown_connections 2024-03-29T11:27:25.212 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.219+0000 7ff3a359e640 1 -- 172.21.15.104:0/977159306 >> 172.21.15.104:0/977159306 conn(0x7ff3a406d0e0 msgr2=0x7ff3a4066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:25.212 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.219+0000 7ff3a359e640 1 -- 172.21.15.104:0/977159306 shutdown_connections 2024-03-29T11:27:25.212 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.219+0000 7ff3a359e640 1 -- 172.21.15.104:0/977159306 wait complete. 2024-03-29T11:27:25.212 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.221+0000 7ff3a359e640 10 monclient: build_initial_monmap 2024-03-29T11:27:25.212 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.221+0000 7ff3a359e640 10 monclient: monmap: 2024-03-29T11:27:25.212 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:27:25.212 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:25.212 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:27:25.213 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:27:25.213 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:25.213 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:25.213 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:27:25.213 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:27:25.213 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:27:25.213 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:25.213 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.221+0000 7ff3a359e640 1 librados: starting msgr at 2024-03-29T11:27:25.213 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.221+0000 7ff3a359e640 1 librados: starting objecter 2024-03-29T11:27:25.213 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.221+0000 7ff3a359e640 1 Processor -- start 2024-03-29T11:27:25.213 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.221+0000 7ff3a359e640 1 -- start start 2024-03-29T11:27:25.213 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.221+0000 7ff3a359e640 1 librados: setting wanted keys 2024-03-29T11:27:25.213 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.221+0000 7ff3a359e640 1 librados: calling monclient init 2024-03-29T11:27:25.213 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.221+0000 7ff3a359e640 10 monclient: init 2024-03-29T11:27:25.213 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.222+0000 7ff3a359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:25.214 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.222+0000 7ff3a359e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:27:25.214 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.223+0000 7ff3a359e640 10 monclient(hunting): picked mon.noname-a con 0x7ff3a40b2e50 addr v1:172.21.15.104:6789/0 2024-03-29T11:27:25.214 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.223+0000 7ff3a359e640 10 monclient(hunting): picked mon.noname-b con 0x7ff3a4143a90 addr v1:172.21.15.119:6789/0 2024-03-29T11:27:25.214 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.223+0000 7ff3a359e640 10 monclient(hunting): picked mon.noname-c con 0x7ff3a4146fd0 addr v1:172.21.15.104:6790/0 2024-03-29T11:27:25.214 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.223+0000 7ff3a359e640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3a40b7bf0 con 0x7ff3a40b2e50 2024-03-29T11:27:25.214 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.223+0000 7ff3a359e640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3a413d5a0 con 0x7ff3a4143a90 2024-03-29T11:27:25.214 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.224+0000 7ff3a359e640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3a413d730 con 0x7ff3a4146fd0 2024-03-29T11:27:25.214 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.224+0000 7ff3a359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:25.214 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.224+0000 7ff3a359e640 10 monclient(hunting): authenticate will time out at 1426.475026s 2024-03-29T11:27:25.215 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.224+0000 7ff3a1d9b640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7ff3a4143a90 0x7ff3a40b5810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:45128/0 (socket says 172.21.15.104:45128) 2024-03-29T11:27:25.216 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.224+0000 7ff3a1d9b640 1 -- 172.21.15.104:0/1577959005 learned_addr learned my addr 172.21.15.104:0/1577959005 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:27:25.216 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.225+0000 7ff392ffd640 10 client.?.objecter ms_handle_connect 0x7ff3a4143a90 2024-03-29T11:27:25.216 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.225+0000 7ff392ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:25.216 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.225+0000 7ff392ffd640 10 client.?.objecter ms_handle_connect 0x7ff3a40b2e50 2024-03-29T11:27:25.216 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.226+0000 7ff392ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:25.222 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.226+0000 7ff392ffd640 1 -- 172.21.15.104:0/1577959005 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1164845683 0 0) 0x7ff3a413d5a0 con 0x7ff3a4143a90 2024-03-29T11:27:25.222 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.226+0000 7ff392ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:25.222 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.226+0000 7ff392ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:25.222 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.226+0000 7ff392ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:25.222 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.226+0000 7ff392ffd640 10 monclient(hunting): my global_id is 4507 2024-03-29T11:27:25.222 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.226+0000 7ff392ffd640 1 -- 172.21.15.104:0/1577959005 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff384004eb0 con 0x7ff3a4143a90 2024-03-29T11:27:25.222 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.226+0000 7ff392ffd640 1 -- 172.21.15.104:0/1577959005 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 245426861 0 0) 0x7ff3a40b7bf0 con 0x7ff3a40b2e50 2024-03-29T11:27:25.222 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.226+0000 7ff392ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:25.222 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.226+0000 7ff392ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:25.222 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.226+0000 7ff392ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:25.222 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.226+0000 7ff392ffd640 10 monclient(hunting): my global_id is 4530 2024-03-29T11:27:25.222 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.226+0000 7ff392ffd640 1 -- 172.21.15.104:0/1577959005 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3a413d5a0 con 0x7ff3a40b2e50 2024-03-29T11:27:25.223 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.226+0000 7ff392ffd640 1 -- 172.21.15.104:0/1577959005 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1322216605 0 0) 0x7ff384004eb0 con 0x7ff3a4143a90 2024-03-29T11:27:25.223 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.227+0000 7ff392ffd640 1 -- 172.21.15.104:0/1577959005 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3a40b7bf0 con 0x7ff3a4143a90 2024-03-29T11:27:25.223 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.227+0000 7ff392ffd640 1 -- 172.21.15.104:0/1577959005 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7ff394001690 con 0x7ff3a4143a90 2024-03-29T11:27:25.223 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.227+0000 7ff392ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:25.223 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.227+0000 7ff392ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:25.223 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.227+0000 7ff392ffd640 10 monclient(hunting): dump: 2024-03-29T11:27:25.223 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:27:25.223 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:27:25.223 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:25.223 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:25.223 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:25.223 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:25.223 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:27:25.223 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:27:25.224 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:27:25.224 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:25.224 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.227+0000 7ff392ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:25.224 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.227+0000 7ff392ffd640 1 -- 172.21.15.104:0/1577959005 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1708575508 0 0) 0x7ff3a413d5a0 con 0x7ff3a40b2e50 2024-03-29T11:27:25.224 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.227+0000 7ff392ffd640 1 -- 172.21.15.104:0/1577959005 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff384004eb0 con 0x7ff3a40b2e50 2024-03-29T11:27:25.224 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.227+0000 7ff392ffd640 1 -- 172.21.15.104:0/1577959005 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7ff398001830 con 0x7ff3a40b2e50 2024-03-29T11:27:25.224 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.227+0000 7ff392ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:25.224 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.227+0000 7ff392ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:25.224 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.227+0000 7ff392ffd640 10 monclient(hunting): dump: 2024-03-29T11:27:25.224 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:27:25.224 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:27:25.224 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:25.224 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:25.224 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:25.224 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:25.225 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:27:25.225 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:27:25.225 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:27:25.225 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:25.225 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.227+0000 7ff392ffd640 1 -- 172.21.15.104:0/1577959005 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4099904036 0 0) 0x7ff3a40b7bf0 con 0x7ff3a4143a90 2024-03-29T11:27:25.225 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.227+0000 7ff392ffd640 1 -- 172.21.15.104:0/1577959005 >> v1:172.21.15.104:6790/0 conn(0x7ff3a4146fd0 legacy=0x7ff3a40b5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:25.225 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.227+0000 7ff392ffd640 1 -- 172.21.15.104:0/1577959005 >> v1:172.21.15.104:6789/0 conn(0x7ff3a40b2e50 legacy=0x7ff3a413ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:25.225 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.227+0000 7ff392ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:27:25.225 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.227+0000 7ff392ffd640 1 monclient: found mon.b 2024-03-29T11:27:25.225 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.227+0000 7ff392ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:25.225 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.228+0000 7ff392ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:27:25.225 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.228+0000 7ff392ffd640 1 -- 172.21.15.104:0/1577959005 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3a413d8c0 con 0x7ff3a4143a90 2024-03-29T11:27:25.225 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.228+0000 7ff392ffd640 10 monclient: _finish_auth 0 2024-03-29T11:27:25.225 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.228+0000 7ff392ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:27:25.225 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.228+0000 7ff392ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:25.226 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.228+0000 7ff3a359e640 5 monclient: authenticate success, global_id 4507 2024-03-29T11:27:25.226 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.228+0000 7ff3a359e640 10 monclient: _renew_subs 2024-03-29T11:27:25.226 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.228+0000 7ff3a359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:27:25.226 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.228+0000 7ff3a359e640 1 -- 172.21.15.104:0/1577959005 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3a4149370 con 0x7ff3a4143a90 2024-03-29T11:27:25.226 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.228+0000 7ff3a359e640 10 client.4507.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:25.226 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.229+0000 7ff3a359e640 10 monclient: _renew_subs 2024-03-29T11:27:25.226 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.229+0000 7ff3a359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:27:25.226 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.229+0000 7ff3a359e640 1 -- 172.21.15.104:0/1577959005 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3a40b7590 con 0x7ff3a4143a90 2024-03-29T11:27:25.226 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.229+0000 7ff3a359e640 1 librados: init done 2024-03-29T11:27:25.226 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.229+0000 7ff392ffd640 1 -- 172.21.15.104:0/1577959005 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3940038a0 con 0x7ff3a4143a90 2024-03-29T11:27:25.226 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.229+0000 7ff392ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:25.226 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.229+0000 7ff392ffd640 1 -- 172.21.15.104:0/1577959005 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7ff394003bf0 con 0x7ff3a4143a90 2024-03-29T11:27:25.226 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.229+0000 7ff392ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:25.226 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.229+0000 7ff392ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:25.227 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.229+0000 7ff3a0d99640 4 set_mon_vals no callback set 2024-03-29T11:27:25.227 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.229+0000 7ff392ffd640 10 monclient: dump: 2024-03-29T11:27:25.227 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:27:25.227 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:27:25.227 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:25.227 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:25.227 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:25.227 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:25.227 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:27:25.227 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:27:25.227 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:27:25.227 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:25.227 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.231+0000 7ff3a359e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:25.227 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.231+0000 7ff3a359e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:25.227 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.232+0000 7ff3a359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:27:25.228 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.232+0000 7ff3a359e640 1 -- 172.21.15.104:0/1577959005 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff388005080 con 0x7ff3a4143a90 2024-03-29T11:27:25.228 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.232+0000 7ff392ffd640 1 -- 172.21.15.104:0/1577959005 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7ff394003d50 con 0x7ff3a4143a90 2024-03-29T11:27:25.228 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.233+0000 7ff392ffd640 1 -- 172.21.15.104:0/1577959005 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7ff394084ac0 con 0x7ff3a4143a90 2024-03-29T11:27:25.228 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.233+0000 7ff392ffd640 10 client.4507.objecter ms_dispatch 0x7ff3a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:25.228 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.233+0000 7ff3a259c640 10 monclient: get_auth_request con 0x7ff38406ddb0 auth_method 0 2024-03-29T11:27:25.228 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.233+0000 7ff392ffd640 3 client.4507.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:25.228 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.233+0000 7ff392ffd640 3 client.4507.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:25.228 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.233+0000 7ff392ffd640 20 client.4507.objecter dump_active .. 0 homeless 2024-03-29T11:27:25.228 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.233+0000 7ff392ffd640 10 client.4507.objecter ms_handle_connect 0x7ff38406ddb0 2024-03-29T11:27:25.231 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.244+0000 7ff392ffd640 1 -- 172.21.15.104:0/1577959005 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff394086e60 con 0x7ff3a4143a90 2024-03-29T11:27:25.232 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.244+0000 7ff392ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:25.232 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.244+0000 7ff392ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:25.551 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.563+0000 7ff3a359e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:25.551 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.563+0000 7ff3a359e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:25.551 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.563+0000 7ff3a359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:27:25.551 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.563+0000 7ff3a359e640 1 -- 172.21.15.104:0/1577959005 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff388005330 con 0x7ff3a4143a90 2024-03-29T11:27:25.552 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.564+0000 7ff392ffd640 1 -- 172.21.15.104:0/1577959005 <== mon.1 v1:172.21.15.119:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7ff3940503a0 con 0x7ff3a4143a90 2024-03-29T11:27:25.552 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.564+0000 7ff392ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:25.552 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.564+0000 7ff392ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:25.553 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:27:25.554 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:25.555 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.568+0000 7ff3a359e640 10 librados: watch_flush enter 2024-03-29T11:27:25.558 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.568+0000 7ff3a359e640 10 librados: watch_flush exit 2024-03-29T11:27:25.559 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.568+0000 7ff3a359e640 20 client.4507.objecter shutdown clearing up homeless session... 2024-03-29T11:27:25.559 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.568+0000 7ff3a359e640 10 client.4507.objecter successfully canceled tick 2024-03-29T11:27:25.559 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.568+0000 7ff3a359e640 1 -- 172.21.15.104:0/1577959005 >> v1:172.21.15.104:6816/3341126660 conn(0x7ff38406ddb0 legacy=0x7ff384070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:25.559 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.568+0000 7ff3a359e640 10 monclient: shutdown 2024-03-29T11:27:25.559 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.568+0000 7ff3a359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:25.559 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.568+0000 7ff3a359e640 1 -- 172.21.15.104:0/1577959005 >> v1:172.21.15.119:6789/0 conn(0x7ff3a4143a90 legacy=0x7ff3a40b5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:25.559 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.571+0000 7ff3a359e640 1 -- 172.21.15.104:0/1577959005 shutdown_connections 2024-03-29T11:27:25.559 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.571+0000 7ff3a359e640 1 -- 172.21.15.104:0/1577959005 >> 172.21.15.104:0/1577959005 conn(0x7ff3a406d0e0 msgr2=0x7ff3a40b2740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:25.560 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.572+0000 7ff3a359e640 1 -- 172.21.15.104:0/1577959005 shutdown_connections 2024-03-29T11:27:25.560 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.572+0000 7ff3a359e640 1 -- 172.21.15.104:0/1577959005 wait complete. 2024-03-29T11:27:25.560 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:25.572+0000 7ff3a359e640 1 librados: shutdown 2024-03-29T11:27:25.580 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:32.388 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:32.538 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.550+0000 7fe0b7ae6640 10 monclient: get_monmap_and_config 2024-03-29T11:27:32.538 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.550+0000 7fe0b7ae6640 10 monclient: build_initial_monmap 2024-03-29T11:27:32.538 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.550+0000 7fe0b7ae6640 10 monclient: monmap: 2024-03-29T11:27:32.538 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:27:32.539 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:27:32.539 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:27:32.551235+0000 2024-03-29T11:27:32.539 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:27:32.551235+0000 2024-03-29T11:27:32.539 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:32.539 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:32.539 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:27:32.539 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:27:32.539 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:27:32.539 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:32.540 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.552+0000 7fe0b7ae6640 1 Processor -- start 2024-03-29T11:27:32.549 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.559+0000 7fe0b7ae6640 1 -- start start 2024-03-29T11:27:32.549 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.559+0000 7fe0b7ae6640 10 monclient: init 2024-03-29T11:27:32.549 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.560+0000 7fe0b7ae6640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:32.549 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.560+0000 7fe0b7ae6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:27:32.549 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.560+0000 7fe0b7ae6640 10 monclient(hunting): picked mon.noname-c con 0x7fe0a8006e30 addr v1:172.21.15.119:6789/0 2024-03-29T11:27:32.549 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.560+0000 7fe0b7ae6640 10 monclient(hunting): picked mon.noname-b con 0x7fe0a8035e00 addr v1:172.21.15.104:6790/0 2024-03-29T11:27:32.549 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.560+0000 7fe0b7ae6640 10 monclient(hunting): picked mon.noname-a con 0x7fe0a80c9670 addr v1:172.21.15.104:6789/0 2024-03-29T11:27:32.549 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.560+0000 7fe0b7ae6640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a8035970 con 0x7fe0a80c9670 2024-03-29T11:27:32.549 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.560+0000 7fe0b7ae6640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a8036280 con 0x7fe0a8006e30 2024-03-29T11:27:32.549 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.560+0000 7fe0b7ae6640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a8035640 con 0x7fe0a8035e00 2024-03-29T11:27:32.549 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.560+0000 7fe0b7ae6640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:32.550 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.560+0000 7fe0b7ae6640 10 monclient(hunting): authenticate will time out at 1433.811693s 2024-03-29T11:27:32.552 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.564+0000 7fe0b5081640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7fe0a8035e00 0x7fe0a80c8f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:36831/0 (socket says 172.21.15.104:36831) 2024-03-29T11:27:32.552 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.564+0000 7fe0b5081640 1 -- 172.21.15.104:0/3744449411 learned_addr learned my addr 172.21.15.104:0/3744449411 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:27:32.557 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.568+0000 7fe0b4880640 1 -- 172.21.15.104:0/3744449411 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3880311762 0 0) 0x7fe0a8035970 con 0x7fe0a80c9670 2024-03-29T11:27:32.557 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.568+0000 7fe0b4880640 10 monclient(hunting): _negotiate 2024-03-29T11:27:32.557 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.568+0000 7fe0b4880640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:32.557 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.568+0000 7fe0b4880640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:32.557 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.568+0000 7fe0b4880640 10 monclient(hunting): my global_id is 4533 2024-03-29T11:27:32.557 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.568+0000 7fe0b4880640 1 -- 172.21.15.104:0/3744449411 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe09c003520 con 0x7fe0a80c9670 2024-03-29T11:27:32.557 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.568+0000 7fe0b4880640 1 -- 172.21.15.104:0/3744449411 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3534373127 0 0) 0x7fe0a8036280 con 0x7fe0a8006e30 2024-03-29T11:27:32.557 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.568+0000 7fe0b4880640 10 monclient(hunting): _negotiate 2024-03-29T11:27:32.557 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.568+0000 7fe0b4880640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:32.557 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.568+0000 7fe0b4880640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:32.557 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.568+0000 7fe0b4880640 10 monclient(hunting): my global_id is 4510 2024-03-29T11:27:32.558 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.568+0000 7fe0b4880640 1 -- 172.21.15.104:0/3744449411 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0a8035970 con 0x7fe0a8006e30 2024-03-29T11:27:32.558 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.568+0000 7fe0b4880640 1 -- 172.21.15.104:0/3744449411 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2425472754 0 0) 0x7fe0a8035640 con 0x7fe0a8035e00 2024-03-29T11:27:32.558 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.568+0000 7fe0b4880640 10 monclient(hunting): _negotiate 2024-03-29T11:27:32.558 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.568+0000 7fe0b4880640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:32.558 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.568+0000 7fe0b4880640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:32.558 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.568+0000 7fe0b4880640 10 monclient(hunting): my global_id is 4541 2024-03-29T11:27:32.558 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.568+0000 7fe0b4880640 1 -- 172.21.15.104:0/3744449411 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0a8036280 con 0x7fe0a8035e00 2024-03-29T11:27:32.559 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.569+0000 7fe0b4880640 1 -- 172.21.15.104:0/3744449411 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3615661061 0 0) 0x7fe0a8036280 con 0x7fe0a8035e00 2024-03-29T11:27:32.559 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.569+0000 7fe0b4880640 1 -- 172.21.15.104:0/3744449411 >> v1:172.21.15.119:6789/0 conn(0x7fe0a8006e30 legacy=0x7fe0a80cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:32.559 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.569+0000 7fe0b4880640 1 -- 172.21.15.104:0/3744449411 >> v1:172.21.15.104:6789/0 conn(0x7fe0a80c9670 legacy=0x7fe0a80d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:32.559 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.569+0000 7fe0b4880640 10 monclient: _finish_hunting 0 2024-03-29T11:27:32.559 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.569+0000 7fe0b4880640 1 monclient: found mon.noname-b 2024-03-29T11:27:32.559 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.569+0000 7fe0b4880640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:32.559 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.569+0000 7fe0b4880640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.104:6790/0 2024-03-29T11:27:32.559 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.569+0000 7fe0b4880640 1 -- 172.21.15.104:0/3744449411 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a8034e20 con 0x7fe0a8035e00 2024-03-29T11:27:32.559 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.569+0000 7fe0b4880640 10 monclient: _finish_auth 0 2024-03-29T11:27:32.559 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.569+0000 7fe0b4880640 10 monclient: _check_auth_tickets 2024-03-29T11:27:32.559 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.569+0000 7fe0b4880640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:32.559 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.569+0000 7fe0b4880640 1 -- 172.21.15.104:0/3744449411 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fe0a00015b0 con 0x7fe0a8035e00 2024-03-29T11:27:32.559 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.569+0000 7fe0b4880640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:32.560 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.569+0000 7fe0b4880640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:32.560 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.569+0000 7fe0b4880640 10 monclient: dump: 2024-03-29T11:27:32.560 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:27:32.560 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:27:32.560 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:32.560 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:32.560 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:32.560 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:32.560 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:27:32.560 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:27:32.560 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:27:32.560 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:32.560 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.569+0000 7fe0b4880640 1 -- 172.21.15.104:0/3744449411 <== mon.2 v1:172.21.15.104:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0a0001780 con 0x7fe0a8035e00 2024-03-29T11:27:32.560 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.569+0000 7fe0b4880640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:32.561 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.569+0000 7fe0b4880640 1 -- 172.21.15.104:0/3744449411 <== mon.2 v1:172.21.15.104:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fe0a0001b10 con 0x7fe0a8035e00 2024-03-29T11:27:32.561 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.569+0000 7fe0b4880640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:32.561 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.569+0000 7fe0b4880640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:32.561 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.569+0000 7fe0b4880640 10 monclient: dump: 2024-03-29T11:27:32.561 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:27:32.561 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:27:32.561 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:32.561 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:32.561 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:32.561 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:32.561 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:27:32.561 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:27:32.561 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:27:32.561 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:32.562 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.571+0000 7fe0b7ae6640 5 monclient: authenticate success, global_id 4541 2024-03-29T11:27:32.562 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.571+0000 7fe0b7ae6640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:32.562 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.571+0000 7fe0b7ae6640 4 set_mon_vals no callback set 2024-03-29T11:27:32.562 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.572+0000 7fe0b7ae6640 10 monclient: shutdown 2024-03-29T11:27:32.562 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.572+0000 7fe0b7ae6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:32.563 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.572+0000 7fe0b7ae6640 1 -- 172.21.15.104:0/3744449411 >> v1:172.21.15.104:6790/0 conn(0x7fe0a8035e00 legacy=0x7fe0a80c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:32.563 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.573+0000 7fe0b7ae6640 1 -- 172.21.15.104:0/3744449411 shutdown_connections 2024-03-29T11:27:32.563 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.573+0000 7fe0b7ae6640 1 -- 172.21.15.104:0/3744449411 >> 172.21.15.104:0/3744449411 conn(0x7fe0a8005c00 msgr2=0x7fe0a8006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:32.563 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.573+0000 7fe0b7ae6640 1 -- 172.21.15.104:0/3744449411 shutdown_connections 2024-03-29T11:27:32.565 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.577+0000 7fe0b7ae6640 1 -- 172.21.15.104:0/3744449411 wait complete. 2024-03-29T11:27:32.566 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.578+0000 7fe0b7ae6640 10 monclient: build_initial_monmap 2024-03-29T11:27:32.566 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.578+0000 7fe0b7ae6640 10 monclient: monmap: 2024-03-29T11:27:32.566 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:27:32.566 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:32.566 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:27:32.566 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:27:32.566 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:32.566 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:32.566 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:27:32.566 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:27:32.567 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:27:32.567 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:32.567 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.578+0000 7fe0b7ae6640 1 librados: starting msgr at 2024-03-29T11:27:32.567 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.578+0000 7fe0b7ae6640 1 librados: starting objecter 2024-03-29T11:27:32.567 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.578+0000 7fe0b7ae6640 1 Processor -- start 2024-03-29T11:27:32.568 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.578+0000 7fe0b7ae6640 1 -- start start 2024-03-29T11:27:32.568 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.578+0000 7fe0b7ae6640 1 librados: setting wanted keys 2024-03-29T11:27:32.569 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.578+0000 7fe0b7ae6640 1 librados: calling monclient init 2024-03-29T11:27:32.569 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.578+0000 7fe0b7ae6640 10 monclient: init 2024-03-29T11:27:32.569 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.579+0000 7fe0b7ae6640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:32.569 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.579+0000 7fe0b7ae6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:27:32.569 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.579+0000 7fe0b7ae6640 10 monclient(hunting): picked mon.noname-a con 0x7fe0a8006e30 addr v1:172.21.15.104:6789/0 2024-03-29T11:27:32.569 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.579+0000 7fe0b7ae6640 10 monclient(hunting): picked mon.noname-b con 0x7fe0a80c9670 addr v1:172.21.15.119:6789/0 2024-03-29T11:27:32.569 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.579+0000 7fe0b7ae6640 10 monclient(hunting): picked mon.noname-c con 0x7fe0a80cef20 addr v1:172.21.15.104:6790/0 2024-03-29T11:27:32.569 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.579+0000 7fe0b7ae6640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a8030230 con 0x7fe0a8006e30 2024-03-29T11:27:32.569 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.579+0000 7fe0b7ae6640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a8160e10 con 0x7fe0a80c9670 2024-03-29T11:27:32.569 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.579+0000 7fe0b7ae6640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a80ce820 con 0x7fe0a80cef20 2024-03-29T11:27:32.569 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.579+0000 7fe0b7ae6640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:32.569 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.579+0000 7fe0b7ae6640 10 monclient(hunting): authenticate will time out at 1433.830419s 2024-03-29T11:27:32.574 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.586+0000 7fe0b5081640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fe0a80c9670 0x7fe0a81612f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:51927/0 (socket says 172.21.15.104:51927) 2024-03-29T11:27:32.574 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.586+0000 7fe0b5081640 1 -- 172.21.15.104:0/3477283073 learned_addr learned my addr 172.21.15.104:0/3477283073 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:27:32.576 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.588+0000 7fe0a67fc640 10 client.?.objecter ms_handle_connect 0x7fe0a80c9670 2024-03-29T11:27:32.576 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.588+0000 7fe0a67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:32.577 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.589+0000 7fe0a67fc640 1 -- 172.21.15.104:0/3477283073 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4266670807 0 0) 0x7fe0a8160e10 con 0x7fe0a80c9670 2024-03-29T11:27:32.577 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.589+0000 7fe0a67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:27:32.577 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.589+0000 7fe0a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:32.577 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.589+0000 7fe0a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:32.577 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.589+0000 7fe0a67fc640 10 monclient(hunting): my global_id is 4513 2024-03-29T11:27:32.577 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.589+0000 7fe0a67fc640 1 -- 172.21.15.104:0/3477283073 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe094004eb0 con 0x7fe0a80c9670 2024-03-29T11:27:32.579 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.591+0000 7fe0a67fc640 10 client.?.objecter ms_handle_connect 0x7fe0a80cef20 2024-03-29T11:27:32.580 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.591+0000 7fe0a67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:32.580 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.591+0000 7fe0a67fc640 10 client.?.objecter ms_handle_connect 0x7fe0a8006e30 2024-03-29T11:27:32.580 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.591+0000 7fe0a67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:32.580 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.591+0000 7fe0a67fc640 1 -- 172.21.15.104:0/3477283073 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2315222289 0 0) 0x7fe0a80ce820 con 0x7fe0a80cef20 2024-03-29T11:27:32.580 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.592+0000 7fe0a67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:27:32.580 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.592+0000 7fe0a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:32.580 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.592+0000 7fe0a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:32.580 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.592+0000 7fe0a67fc640 10 monclient(hunting): my global_id is 4544 2024-03-29T11:27:32.581 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.592+0000 7fe0a67fc640 1 -- 172.21.15.104:0/3477283073 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0a8160e10 con 0x7fe0a80cef20 2024-03-29T11:27:32.582 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.594+0000 7fe0a67fc640 1 -- 172.21.15.104:0/3477283073 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2924787811 0 0) 0x7fe0a8030230 con 0x7fe0a8006e30 2024-03-29T11:27:32.582 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.594+0000 7fe0a67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:27:32.582 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.594+0000 7fe0a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:32.582 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.594+0000 7fe0a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:32.582 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.594+0000 7fe0a67fc640 10 monclient(hunting): my global_id is 4536 2024-03-29T11:27:32.582 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.594+0000 7fe0a67fc640 1 -- 172.21.15.104:0/3477283073 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0a80ce820 con 0x7fe0a8006e30 2024-03-29T11:27:32.582 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.594+0000 7fe0a67fc640 1 -- 172.21.15.104:0/3477283073 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2322275638 0 0) 0x7fe094004eb0 con 0x7fe0a80c9670 2024-03-29T11:27:32.582 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.594+0000 7fe0a67fc640 1 -- 172.21.15.104:0/3477283073 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0a8030230 con 0x7fe0a80c9670 2024-03-29T11:27:32.582 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.594+0000 7fe0a67fc640 1 -- 172.21.15.104:0/3477283073 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fe0a00014b0 con 0x7fe0a80c9670 2024-03-29T11:27:32.582 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.594+0000 7fe0a67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:32.582 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.594+0000 7fe0a67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:32.582 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.594+0000 7fe0a67fc640 10 monclient(hunting): dump: 2024-03-29T11:27:32.583 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:27:32.583 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:27:32.583 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:32.583 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:32.583 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:32.583 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:32.583 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:27:32.583 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:27:32.583 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:27:32.583 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:32.583 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.594+0000 7fe0a67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:32.585 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.594+0000 7fe0a67fc640 1 -- 172.21.15.104:0/3477283073 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3282868085 0 0) 0x7fe0a80ce820 con 0x7fe0a8006e30 2024-03-29T11:27:32.585 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.594+0000 7fe0a67fc640 1 -- 172.21.15.104:0/3477283073 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe094004eb0 con 0x7fe0a8006e30 2024-03-29T11:27:32.585 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.594+0000 7fe0a67fc640 1 -- 172.21.15.104:0/3477283073 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fe0ac001780 con 0x7fe0a8006e30 2024-03-29T11:27:32.585 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.594+0000 7fe0a67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:32.585 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.594+0000 7fe0a67fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:32.585 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.594+0000 7fe0a67fc640 10 monclient(hunting): dump: 2024-03-29T11:27:32.585 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:27:32.585 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:27:32.585 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:32.585 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:32.586 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:32.586 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:32.586 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:27:32.586 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:27:32.586 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:27:32.586 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:32.586 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.594+0000 7fe0a67fc640 1 -- 172.21.15.104:0/3477283073 <== mon.0 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3290011055 0 0) 0x7fe094004eb0 con 0x7fe0a8006e30 2024-03-29T11:27:32.586 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.594+0000 7fe0a67fc640 1 -- 172.21.15.104:0/3477283073 >> v1:172.21.15.104:6790/0 conn(0x7fe0a80cef20 legacy=0x7fe0a80cf680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:32.586 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.594+0000 7fe0a67fc640 1 -- 172.21.15.104:0/3477283073 >> v1:172.21.15.119:6789/0 conn(0x7fe0a80c9670 legacy=0x7fe0a81612f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:32.586 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.594+0000 7fe0a67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:27:32.586 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.594+0000 7fe0a67fc640 1 monclient: found mon.a 2024-03-29T11:27:32.586 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.594+0000 7fe0a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:32.586 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.594+0000 7fe0a67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:27:32.587 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.595+0000 7fe0a67fc640 1 -- 172.21.15.104:0/3477283073 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a8161a00 con 0x7fe0a8006e30 2024-03-29T11:27:32.587 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.595+0000 7fe0a67fc640 10 monclient: _finish_auth 0 2024-03-29T11:27:32.587 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.595+0000 7fe0a67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:27:32.587 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.595+0000 7fe0a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:32.587 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.595+0000 7fe0b7ae6640 5 monclient: authenticate success, global_id 4536 2024-03-29T11:27:32.587 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.595+0000 7fe0b7ae6640 10 monclient: _renew_subs 2024-03-29T11:27:32.587 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.595+0000 7fe0b7ae6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:27:32.587 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.595+0000 7fe0b7ae6640 1 -- 172.21.15.104:0/3477283073 --> v1:172.21.15.104:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0a80cfd90 con 0x7fe0a8006e30 2024-03-29T11:27:32.587 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.595+0000 7fe0b7ae6640 10 client.4536.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:32.587 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.595+0000 7fe0b7ae6640 10 monclient: _renew_subs 2024-03-29T11:27:32.587 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.595+0000 7fe0b7ae6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:27:32.587 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.595+0000 7fe0b7ae6640 1 -- 172.21.15.104:0/3477283073 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0a80cd1a0 con 0x7fe0a8006e30 2024-03-29T11:27:32.587 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.595+0000 7fe0b7ae6640 1 librados: init done 2024-03-29T11:27:32.587 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.596+0000 7fe0a67fc640 1 -- 172.21.15.104:0/3477283073 <== mon.0 v1:172.21.15.104:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0ac001bf0 con 0x7fe0a8006e30 2024-03-29T11:27:32.587 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.596+0000 7fe0a67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:32.588 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.596+0000 7fe0a67fc640 1 -- 172.21.15.104:0/3477283073 <== mon.0 v1:172.21.15.104:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fe0ac0037b0 con 0x7fe0a8006e30 2024-03-29T11:27:32.588 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.596+0000 7fe0a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:32.588 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.596+0000 7fe0a67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:32.588 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.596+0000 7fe0a67fc640 10 monclient: dump: 2024-03-29T11:27:32.588 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:27:32.588 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:27:32.588 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:32.588 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:32.588 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:32.588 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:32.588 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:27:32.588 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:27:32.588 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:27:32.588 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:32.590 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.597+0000 7fe0a77fe640 4 set_mon_vals no callback set 2024-03-29T11:27:32.590 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.598+0000 7fe0b7ae6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:32.590 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.598+0000 7fe0a67fc640 1 -- 172.21.15.104:0/3477283073 <== mon.0 v1:172.21.15.104:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7fe0ac019780 con 0x7fe0a8006e30 2024-03-29T11:27:32.590 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.598+0000 7fe0b7ae6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:32.590 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.598+0000 7fe0b7ae6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:27:32.590 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.598+0000 7fe0b7ae6640 1 -- 172.21.15.104:0/3477283073 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe098005080 con 0x7fe0a8006e30 2024-03-29T11:27:32.590 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.599+0000 7fe0a67fc640 1 -- 172.21.15.104:0/3477283073 <== mon.0 v1:172.21.15.104:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7fe0ac0836b0 con 0x7fe0a8006e30 2024-03-29T11:27:32.590 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.599+0000 7fe0b5081640 10 monclient: get_auth_request con 0x7fe09406de70 auth_method 0 2024-03-29T11:27:32.590 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.600+0000 7fe0a67fc640 10 client.4536.objecter ms_dispatch 0x7fe0b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:32.591 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.600+0000 7fe0a67fc640 3 client.4536.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:32.591 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.600+0000 7fe0a67fc640 3 client.4536.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:32.591 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.600+0000 7fe0a67fc640 20 client.4536.objecter dump_active .. 0 homeless 2024-03-29T11:27:32.591 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.600+0000 7fe0a67fc640 10 client.4536.objecter ms_handle_connect 0x7fe09406de70 2024-03-29T11:27:32.599 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.610+0000 7fe0a67fc640 1 -- 172.21.15.104:0/3477283073 <== mon.0 v1:172.21.15.104:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe0ac04ef90 con 0x7fe0a8006e30 2024-03-29T11:27:32.599 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.610+0000 7fe0a67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:32.599 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.610+0000 7fe0a67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:32.921 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.933+0000 7fe0b7ae6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:32.921 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.933+0000 7fe0b7ae6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:32.921 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.933+0000 7fe0b7ae6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:27:32.921 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.933+0000 7fe0b7ae6640 1 -- 172.21.15.104:0/3477283073 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe098005330 con 0x7fe0a8006e30 2024-03-29T11:27:32.923 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.934+0000 7fe0a67fc640 1 -- 172.21.15.104:0/3477283073 <== mon.0 v1:172.21.15.104:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7fe0ac050ec0 con 0x7fe0a8006e30 2024-03-29T11:27:32.923 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.934+0000 7fe0a67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:32.923 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.934+0000 7fe0a67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:32.924 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:27:32.924 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:32.931 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.938+0000 7fe0b7ae6640 10 librados: watch_flush enter 2024-03-29T11:27:32.931 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.939+0000 7fe0b7ae6640 10 librados: watch_flush exit 2024-03-29T11:27:32.931 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.939+0000 7fe0b7ae6640 20 client.4536.objecter shutdown clearing up homeless session... 2024-03-29T11:27:32.931 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.939+0000 7fe0b7ae6640 10 client.4536.objecter successfully canceled tick 2024-03-29T11:27:32.931 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.939+0000 7fe0b7ae6640 1 -- 172.21.15.104:0/3477283073 >> v1:172.21.15.104:6816/3341126660 conn(0x7fe09406de70 legacy=0x7fe094070330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:32.931 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.939+0000 7fe0b7ae6640 10 monclient: shutdown 2024-03-29T11:27:32.931 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.939+0000 7fe0b7ae6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:32.931 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.939+0000 7fe0b7ae6640 1 -- 172.21.15.104:0/3477283073 >> v1:172.21.15.104:6789/0 conn(0x7fe0a8006e30 legacy=0x7fe0a802fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:32.931 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.941+0000 7fe0b7ae6640 1 -- 172.21.15.104:0/3477283073 shutdown_connections 2024-03-29T11:27:32.931 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.941+0000 7fe0b7ae6640 1 -- 172.21.15.104:0/3477283073 >> 172.21.15.104:0/3477283073 conn(0x7fe0a8005c00 msgr2=0x7fe0a80c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:32.931 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.942+0000 7fe0b7ae6640 1 -- 172.21.15.104:0/3477283073 shutdown_connections 2024-03-29T11:27:32.932 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.942+0000 7fe0b7ae6640 1 -- 172.21.15.104:0/3477283073 wait complete. 2024-03-29T11:27:32.932 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:32.942+0000 7fe0b7ae6640 1 librados: shutdown 2024-03-29T11:27:32.949 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:39.756 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:39.906 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.917+0000 7f933b625640 10 monclient: get_monmap_and_config 2024-03-29T11:27:39.906 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.917+0000 7f933b625640 10 monclient: build_initial_monmap 2024-03-29T11:27:39.906 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.917+0000 7f933b625640 10 monclient: monmap: 2024-03-29T11:27:39.906 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:27:39.906 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:27:39.906 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:27:39.918561+0000 2024-03-29T11:27:39.906 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:27:39.918561+0000 2024-03-29T11:27:39.906 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:39.906 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:39.906 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:27:39.907 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:27:39.907 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:27:39.907 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:39.907 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.918+0000 7f933b625640 1 Processor -- start 2024-03-29T11:27:39.910 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.919+0000 7f933b625640 1 -- start start 2024-03-29T11:27:39.910 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.919+0000 7f933b625640 10 monclient: init 2024-03-29T11:27:39.910 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.921+0000 7f933b625640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:39.910 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.921+0000 7f933b625640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:27:39.911 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.921+0000 7f933b625640 10 monclient(hunting): picked mon.noname-b con 0x7f9334070120 addr v1:172.21.15.104:6790/0 2024-03-29T11:27:39.911 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.921+0000 7f933b625640 10 monclient(hunting): picked mon.noname-a con 0x7f93340b0420 addr v1:172.21.15.104:6789/0 2024-03-29T11:27:39.911 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.921+0000 7f933b625640 10 monclient(hunting): picked mon.noname-c con 0x7f9334070500 addr v1:172.21.15.119:6789/0 2024-03-29T11:27:39.911 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.921+0000 7f933b625640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9334072090 con 0x7f93340b0420 2024-03-29T11:27:39.911 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.921+0000 7f933b625640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9334072370 con 0x7f9334070500 2024-03-29T11:27:39.911 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.921+0000 7f933b625640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9334072650 con 0x7f9334070120 2024-03-29T11:27:39.911 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.921+0000 7f933b625640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:39.911 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.921+0000 7f933b625640 10 monclient(hunting): authenticate will time out at 1441.172209s 2024-03-29T11:27:39.913 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.925+0000 7f93393c1640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f9334070120 0x7f933413d350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:49249/0 (socket says 172.21.15.104:49249) 2024-03-29T11:27:39.913 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.925+0000 7f93393c1640 1 -- 172.21.15.104:0/1856083182 learned_addr learned my addr 172.21.15.104:0/1856083182 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:27:39.913 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.925+0000 7f9327fff640 1 -- 172.21.15.104:0/1856083182 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3651957289 0 0) 0x7f9334072650 con 0x7f9334070120 2024-03-29T11:27:39.913 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.925+0000 7f9327fff640 10 monclient(hunting): _negotiate 2024-03-29T11:27:39.914 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.925+0000 7f9327fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:39.914 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.925+0000 7f9327fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:39.914 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.925+0000 7f9327fff640 10 monclient(hunting): my global_id is 4547 2024-03-29T11:27:39.914 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.925+0000 7f9327fff640 1 -- 172.21.15.104:0/1856083182 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9320003520 con 0x7f9334070120 2024-03-29T11:27:39.914 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.926+0000 7f9327fff640 1 -- 172.21.15.104:0/1856083182 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1778328250 0 0) 0x7f9320003520 con 0x7f9334070120 2024-03-29T11:27:39.914 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.926+0000 7f9327fff640 1 -- 172.21.15.104:0/1856083182 >> v1:172.21.15.119:6789/0 conn(0x7f9334070500 legacy=0x7f933414b810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:39.917 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.927+0000 7f9327fff640 1 -- 172.21.15.104:0/1856083182 >> v1:172.21.15.104:6789/0 conn(0x7f93340b0420 legacy=0x7f93341405b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:39.917 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.927+0000 7f9327fff640 10 monclient: _finish_hunting 0 2024-03-29T11:27:39.917 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.927+0000 7f9327fff640 1 monclient: found mon.noname-b 2024-03-29T11:27:39.917 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.927+0000 7f9327fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:39.917 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.927+0000 7f9327fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.104:6790/0 2024-03-29T11:27:39.917 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.927+0000 7f9327fff640 1 -- 172.21.15.104:0/1856083182 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9334140090 con 0x7f9334070120 2024-03-29T11:27:39.917 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.927+0000 7f9327fff640 10 monclient: _finish_auth 0 2024-03-29T11:27:39.917 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.927+0000 7f9327fff640 10 monclient: _check_auth_tickets 2024-03-29T11:27:39.917 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.927+0000 7f9327fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:39.917 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.927+0000 7f933b625640 5 monclient: authenticate success, global_id 4547 2024-03-29T11:27:39.917 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.927+0000 7f933b625640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:39.917 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.928+0000 7f9327fff640 1 -- 172.21.15.104:0/1856083182 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f9330001690 con 0x7f9334070120 2024-03-29T11:27:39.917 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.928+0000 7f9327fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:39.918 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.928+0000 7f9327fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:39.918 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.928+0000 7f9327fff640 10 monclient: dump: 2024-03-29T11:27:39.918 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:27:39.918 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:27:39.918 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:39.918 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:39.918 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:39.918 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:39.918 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:27:39.918 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:27:39.918 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:27:39.918 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:39.918 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.928+0000 7f933b625640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:39.918 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.928+0000 7f9327fff640 1 -- 172.21.15.104:0/1856083182 <== mon.2 v1:172.21.15.104:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f93300018b0 con 0x7f9334070120 2024-03-29T11:27:39.918 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.928+0000 7f9327fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:39.919 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.928+0000 7f933b625640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:39.919 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.928+0000 7f933b625640 4 set_mon_vals no callback set 2024-03-29T11:27:39.920 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.929+0000 7f933b625640 10 monclient: shutdown 2024-03-29T11:27:39.920 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.929+0000 7f933b625640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:39.920 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.929+0000 7f933b625640 1 -- 172.21.15.104:0/1856083182 >> v1:172.21.15.104:6790/0 conn(0x7f9334070120 legacy=0x7f933413d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:39.920 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.929+0000 7f933b625640 1 -- 172.21.15.104:0/1856083182 shutdown_connections 2024-03-29T11:27:39.920 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.929+0000 7f933b625640 1 -- 172.21.15.104:0/1856083182 >> 172.21.15.104:0/1856083182 conn(0x7f9334067500 msgr2=0x7f93340b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:39.922 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.933+0000 7f933b625640 1 -- 172.21.15.104:0/1856083182 shutdown_connections 2024-03-29T11:27:39.923 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.933+0000 7f933b625640 1 -- 172.21.15.104:0/1856083182 wait complete. 2024-03-29T11:27:39.923 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.934+0000 7f933b625640 10 monclient: build_initial_monmap 2024-03-29T11:27:39.923 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.934+0000 7f933b625640 10 monclient: monmap: 2024-03-29T11:27:39.923 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:27:39.923 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:39.923 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:27:39.923 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:27:39.924 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:39.924 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:39.924 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:27:39.924 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:27:39.924 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:27:39.924 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:39.924 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.934+0000 7f933b625640 1 librados: starting msgr at 2024-03-29T11:27:39.924 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.934+0000 7f933b625640 1 librados: starting objecter 2024-03-29T11:27:39.924 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.934+0000 7f933b625640 1 Processor -- start 2024-03-29T11:27:39.926 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.935+0000 7f933b625640 1 -- start start 2024-03-29T11:27:39.926 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.935+0000 7f933b625640 1 librados: setting wanted keys 2024-03-29T11:27:39.926 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.935+0000 7f933b625640 1 librados: calling monclient init 2024-03-29T11:27:39.926 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.935+0000 7f933b625640 10 monclient: init 2024-03-29T11:27:39.926 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.936+0000 7f933b625640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:39.926 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.936+0000 7f933b625640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:27:39.927 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.936+0000 7f933b625640 10 monclient(hunting): picked mon.noname-a con 0x7f9334070120 addr v1:172.21.15.104:6789/0 2024-03-29T11:27:39.927 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.936+0000 7f933b625640 10 monclient(hunting): picked mon.noname-b con 0x7f9334070500 addr v1:172.21.15.119:6789/0 2024-03-29T11:27:39.927 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.936+0000 7f933b625640 10 monclient(hunting): picked mon.noname-c con 0x7f93340ab120 addr v1:172.21.15.104:6790/0 2024-03-29T11:27:39.927 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.936+0000 7f933b625640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93341414e0 con 0x7f9334070120 2024-03-29T11:27:39.927 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.936+0000 7f933b625640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93340aea60 con 0x7f9334070500 2024-03-29T11:27:39.927 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.936+0000 7f933b625640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93341e79e0 con 0x7f93340ab120 2024-03-29T11:27:39.927 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.936+0000 7f933b625640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:39.927 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.936+0000 7f933b625640 10 monclient(hunting): authenticate will time out at 1441.187175s 2024-03-29T11:27:39.930 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.940+0000 7f93393c1640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f9334070120 0x7f93340ae350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:33429/0 (socket says 172.21.15.104:33429) 2024-03-29T11:27:39.930 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.941+0000 7f93393c1640 1 -- 172.21.15.104:0/1002134502 learned_addr learned my addr 172.21.15.104:0/1002134502 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:27:39.930 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.941+0000 7f9325ffb640 10 client.?.objecter ms_handle_connect 0x7f9334070120 2024-03-29T11:27:39.930 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.941+0000 7f9325ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:39.930 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.941+0000 7f9325ffb640 1 -- 172.21.15.104:0/1002134502 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1404103056 0 0) 0x7f93341414e0 con 0x7f9334070120 2024-03-29T11:27:39.931 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.941+0000 7f9325ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:39.931 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.941+0000 7f9325ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:39.931 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.941+0000 7f9325ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:39.931 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.941+0000 7f9325ffb640 10 monclient(hunting): my global_id is 4542 2024-03-29T11:27:39.931 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.941+0000 7f9325ffb640 1 -- 172.21.15.104:0/1002134502 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9318004eb0 con 0x7f9334070120 2024-03-29T11:27:39.931 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.942+0000 7f9325ffb640 1 -- 172.21.15.104:0/1002134502 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2656654245 0 0) 0x7f9318004eb0 con 0x7f9334070120 2024-03-29T11:27:39.931 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.942+0000 7f9325ffb640 1 -- 172.21.15.104:0/1002134502 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f93341414e0 con 0x7f9334070120 2024-03-29T11:27:39.931 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.942+0000 7f9325ffb640 1 -- 172.21.15.104:0/1002134502 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f9330004ae0 con 0x7f9334070120 2024-03-29T11:27:39.931 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.942+0000 7f9325ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:39.931 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.942+0000 7f9325ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:39.931 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.942+0000 7f9325ffb640 10 monclient(hunting): dump: 2024-03-29T11:27:39.931 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:27:39.931 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:27:39.931 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:39.931 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:39.932 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:39.932 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:39.932 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:27:39.932 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:27:39.932 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:27:39.932 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:39.932 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.942+0000 7f9325ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:39.935 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.943+0000 7f9325ffb640 1 -- 172.21.15.104:0/1002134502 <== mon.0 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3130725421 0 0) 0x7f93341414e0 con 0x7f9334070120 2024-03-29T11:27:39.935 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.943+0000 7f9325ffb640 1 -- 172.21.15.104:0/1002134502 >> v1:172.21.15.104:6790/0 conn(0x7f93340ab120 legacy=0x7f93341e14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:39.936 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.943+0000 7f9325ffb640 1 -- 172.21.15.104:0/1002134502 >> v1:172.21.15.119:6789/0 conn(0x7f9334070500 legacy=0x7f93341e7f00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:27:39.936 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.943+0000 7f9325ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:27:39.936 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.943+0000 7f9325ffb640 1 monclient: found mon.a 2024-03-29T11:27:39.936 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.943+0000 7f9325ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:39.936 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.943+0000 7f9325ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:27:39.936 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.943+0000 7f9325ffb640 1 -- 172.21.15.104:0/1002134502 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93341e8610 con 0x7f9334070120 2024-03-29T11:27:39.936 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.943+0000 7f9325ffb640 10 monclient: _finish_auth 0 2024-03-29T11:27:39.936 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.943+0000 7f9325ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:27:39.936 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.943+0000 7f9325ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:39.936 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.944+0000 7f9325ffb640 1 -- 172.21.15.104:0/1002134502 <== mon.0 v1:172.21.15.104:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9330004ef0 con 0x7f9334070120 2024-03-29T11:27:39.936 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.944+0000 7f9325ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:39.936 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.944+0000 7f9325ffb640 1 -- 172.21.15.104:0/1002134502 <== mon.0 v1:172.21.15.104:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f9330005300 con 0x7f9334070120 2024-03-29T11:27:39.936 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.944+0000 7f9325ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:39.936 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.944+0000 7f9325ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:39.936 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.944+0000 7f9325ffb640 10 monclient: dump: 2024-03-29T11:27:39.937 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:27:39.937 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:27:39.937 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:39.937 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:39.937 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:39.937 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:39.937 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:27:39.937 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:27:39.937 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:27:39.937 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:39.937 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.944+0000 7f93277fe640 4 set_mon_vals no callback set 2024-03-29T11:27:39.937 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.945+0000 7f933b625640 5 monclient: authenticate success, global_id 4542 2024-03-29T11:27:39.937 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.945+0000 7f933b625640 10 monclient: _renew_subs 2024-03-29T11:27:39.937 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.945+0000 7f933b625640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:27:39.937 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.945+0000 7f933b625640 1 -- 172.21.15.104:0/1002134502 --> v1:172.21.15.104:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93340adb90 con 0x7f9334070120 2024-03-29T11:27:39.938 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.945+0000 7f933b625640 10 client.4542.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:39.938 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.945+0000 7f933b625640 10 monclient: _renew_subs 2024-03-29T11:27:39.938 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.945+0000 7f933b625640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:27:39.938 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.945+0000 7f933b625640 1 -- 172.21.15.104:0/1002134502 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f93341e1bb0 con 0x7f9334070120 2024-03-29T11:27:39.938 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.945+0000 7f933b625640 1 librados: init done 2024-03-29T11:27:39.938 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.947+0000 7f933b625640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:39.938 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.948+0000 7f933b625640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:39.938 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.948+0000 7f933b625640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:27:39.938 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.948+0000 7f933b625640 1 -- 172.21.15.104:0/1002134502 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93341e7840 con 0x7f9334070120 2024-03-29T11:27:39.941 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.951+0000 7f9325ffb640 1 -- 172.21.15.104:0/1002134502 <== mon.0 v1:172.21.15.104:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f9330001ae0 con 0x7f9334070120 2024-03-29T11:27:39.941 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.951+0000 7f9325ffb640 1 -- 172.21.15.104:0/1002134502 <== mon.0 v1:172.21.15.104:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f9330084eb0 con 0x7f9334070120 2024-03-29T11:27:39.941 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.951+0000 7f9325ffb640 10 client.4542.objecter ms_dispatch 0x7f9334000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:39.941 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.951+0000 7f9325ffb640 3 client.4542.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:39.941 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.951+0000 7f9325ffb640 3 client.4542.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:39.941 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.951+0000 7f9338bc0640 10 monclient: get_auth_request con 0x7f931806dd70 auth_method 0 2024-03-29T11:27:39.942 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.951+0000 7f9325ffb640 20 client.4542.objecter dump_active .. 0 homeless 2024-03-29T11:27:39.942 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.952+0000 7f9325ffb640 10 client.4542.objecter ms_handle_connect 0x7f931806dd70 2024-03-29T11:27:39.945 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.956+0000 7f9325ffb640 1 -- 172.21.15.104:0/1002134502 <== mon.0 v1:172.21.15.104:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f93341e7840 con 0x7f9334070120 2024-03-29T11:27:39.945 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.956+0000 7f9325ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:39.945 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:39.956+0000 7f9325ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:40.272 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:40.283+0000 7f933b625640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:40.272 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:40.283+0000 7f933b625640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:40.272 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:40.283+0000 7f933b625640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:27:40.273 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:40.283+0000 7f933b625640 1 -- 172.21.15.104:0/1002134502 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f93341e7840 con 0x7f9334070120 2024-03-29T11:27:40.273 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:40.284+0000 7f9325ffb640 1 -- 172.21.15.104:0/1002134502 <== mon.0 v1:172.21.15.104:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f93341e7840 con 0x7f9334070120 2024-03-29T11:27:40.273 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:40.284+0000 7f9325ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:40.273 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:40.284+0000 7f9325ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:40.273 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:27:40.274 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:40.277 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:40.289+0000 7f933b625640 10 librados: watch_flush enter 2024-03-29T11:27:40.278 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:40.289+0000 7f933b625640 10 librados: watch_flush exit 2024-03-29T11:27:40.278 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:40.289+0000 7f933b625640 20 client.4542.objecter shutdown clearing up homeless session... 2024-03-29T11:27:40.278 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:40.289+0000 7f933b625640 10 client.4542.objecter successfully canceled tick 2024-03-29T11:27:40.278 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:40.290+0000 7f933b625640 1 -- 172.21.15.104:0/1002134502 >> v1:172.21.15.104:6816/3341126660 conn(0x7f931806dd70 legacy=0x7f9318070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:40.278 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:40.290+0000 7f933b625640 10 monclient: shutdown 2024-03-29T11:27:40.278 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:40.290+0000 7f933b625640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:40.278 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:40.290+0000 7f933b625640 1 -- 172.21.15.104:0/1002134502 >> v1:172.21.15.104:6789/0 conn(0x7f9334070120 legacy=0x7f93340ae350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:40.280 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:40.292+0000 7f933b625640 1 -- 172.21.15.104:0/1002134502 shutdown_connections 2024-03-29T11:27:40.280 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:40.292+0000 7f933b625640 1 -- 172.21.15.104:0/1002134502 >> 172.21.15.104:0/1002134502 conn(0x7f9334067500 msgr2=0x7f933413d120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:40.280 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:40.292+0000 7f933b625640 1 -- 172.21.15.104:0/1002134502 shutdown_connections 2024-03-29T11:27:40.280 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:40.292+0000 7f933b625640 1 -- 172.21.15.104:0/1002134502 wait complete. 2024-03-29T11:27:40.281 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:40.293+0000 7f933b625640 1 librados: shutdown 2024-03-29T11:27:40.298 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:47.106 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:47.269 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.280+0000 7f9c1c257640 10 monclient: get_monmap_and_config 2024-03-29T11:27:47.269 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.280+0000 7f9c1c257640 10 monclient: build_initial_monmap 2024-03-29T11:27:47.269 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.280+0000 7f9c1c257640 10 monclient: monmap: 2024-03-29T11:27:47.269 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:27:47.269 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:27:47.270 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:27:47.280974+0000 2024-03-29T11:27:47.270 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:27:47.280974+0000 2024-03-29T11:27:47.270 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:47.270 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:47.270 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:27:47.270 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:27:47.270 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:27:47.270 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:47.270 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.281+0000 7f9c1c257640 1 Processor -- start 2024-03-29T11:27:47.271 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.281+0000 7f9c1c257640 1 -- start start 2024-03-29T11:27:47.271 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.281+0000 7f9c1c257640 10 monclient: init 2024-03-29T11:27:47.271 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.282+0000 7f9c1c257640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:47.272 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.282+0000 7f9c1c257640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:27:47.272 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.282+0000 7f9c1c257640 10 monclient(hunting): picked mon.noname-c con 0x7f9c140b0770 addr v1:172.21.15.119:6789/0 2024-03-29T11:27:47.272 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.282+0000 7f9c1c257640 10 monclient(hunting): picked mon.noname-b con 0x7f9c1414a700 addr v1:172.21.15.104:6790/0 2024-03-29T11:27:47.272 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.282+0000 7f9c1c257640 10 monclient(hunting): picked mon.noname-a con 0x7f9c140ac5a0 addr v1:172.21.15.104:6789/0 2024-03-29T11:27:47.272 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.282+0000 7f9c1c257640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c140aba70 con 0x7f9c140ac5a0 2024-03-29T11:27:47.272 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.282+0000 7f9c1c257640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c140aecb0 con 0x7f9c140b0770 2024-03-29T11:27:47.272 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.282+0000 7f9c1c257640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c140af150 con 0x7f9c1414a700 2024-03-29T11:27:47.272 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.282+0000 7f9c1c257640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:47.272 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.282+0000 7f9c1c257640 10 monclient(hunting): authenticate will time out at 1448.533512s 2024-03-29T11:27:47.272 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.282+0000 7f9c19ff3640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f9c140b0770 0x7f9c14149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:49084/0 (socket says 172.21.15.104:49084) 2024-03-29T11:27:47.272 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.282+0000 7f9c19ff3640 1 -- 172.21.15.104:0/1269205616 learned_addr learned my addr 172.21.15.104:0/1269205616 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:27:47.272 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.283+0000 7f9c18ff1640 1 -- 172.21.15.104:0/1269205616 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3613616500 0 0) 0x7f9c140aecb0 con 0x7f9c140b0770 2024-03-29T11:27:47.272 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.283+0000 7f9c18ff1640 10 monclient(hunting): _negotiate 2024-03-29T11:27:47.273 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.283+0000 7f9c18ff1640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:47.273 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.283+0000 7f9c18ff1640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:47.273 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.283+0000 7f9c18ff1640 10 monclient(hunting): my global_id is 4516 2024-03-29T11:27:47.273 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.283+0000 7f9c18ff1640 1 -- 172.21.15.104:0/1269205616 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c00003520 con 0x7f9c140b0770 2024-03-29T11:27:47.274 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.283+0000 7f9c18ff1640 1 -- 172.21.15.104:0/1269205616 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2382806002 0 0) 0x7f9c00003520 con 0x7f9c140b0770 2024-03-29T11:27:47.275 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.283+0000 7f9c18ff1640 1 -- 172.21.15.104:0/1269205616 >> v1:172.21.15.104:6790/0 conn(0x7f9c1414a700 legacy=0x7f9c140abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:47.275 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.283+0000 7f9c18ff1640 1 -- 172.21.15.104:0/1269205616 >> v1:172.21.15.104:6789/0 conn(0x7f9c140ac5a0 legacy=0x7f9c14151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:47.275 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.283+0000 7f9c18ff1640 10 monclient: _finish_hunting 0 2024-03-29T11:27:47.275 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.284+0000 7f9c18ff1640 1 monclient: found mon.noname-c 2024-03-29T11:27:47.275 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.284+0000 7f9c18ff1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:47.275 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.284+0000 7f9c18ff1640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.119:6789/0 2024-03-29T11:27:47.275 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.284+0000 7f9c18ff1640 1 -- 172.21.15.104:0/1269205616 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c140ac9d0 con 0x7f9c140b0770 2024-03-29T11:27:47.275 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.284+0000 7f9c18ff1640 10 monclient: _finish_auth 0 2024-03-29T11:27:47.275 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.284+0000 7f9c18ff1640 10 monclient: _check_auth_tickets 2024-03-29T11:27:47.275 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.284+0000 7f9c18ff1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:47.275 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.284+0000 7f9c1c257640 5 monclient: authenticate success, global_id 4516 2024-03-29T11:27:47.275 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.284+0000 7f9c1c257640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:47.276 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.284+0000 7f9c18ff1640 1 -- 172.21.15.104:0/1269205616 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f9c040016b0 con 0x7f9c140b0770 2024-03-29T11:27:47.276 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.284+0000 7f9c18ff1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:47.276 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.284+0000 7f9c18ff1640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:47.276 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.284+0000 7f9c18ff1640 10 monclient: dump: 2024-03-29T11:27:47.276 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:27:47.276 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:27:47.276 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:47.276 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:47.276 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:47.276 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:47.276 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:27:47.276 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:27:47.276 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:27:47.277 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:47.277 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.284+0000 7f9c1c257640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:47.277 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.284+0000 7f9c18ff1640 1 -- 172.21.15.104:0/1269205616 <== mon.1 v1:172.21.15.119:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c04001810 con 0x7f9c140b0770 2024-03-29T11:27:47.277 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.284+0000 7f9c18ff1640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:47.277 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.284+0000 7f9c1c257640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:47.277 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.284+0000 7f9c1c257640 4 set_mon_vals no callback set 2024-03-29T11:27:47.277 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.284+0000 7f9c18ff1640 1 -- 172.21.15.104:0/1269205616 <== mon.1 v1:172.21.15.119:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f9c04001c20 con 0x7f9c140b0770 2024-03-29T11:27:47.277 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.284+0000 7f9c18ff1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:47.277 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.284+0000 7f9c18ff1640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:47.277 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.284+0000 7f9c18ff1640 10 monclient: dump: 2024-03-29T11:27:47.277 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:27:47.277 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:27:47.277 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:47.277 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:47.278 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:47.278 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:47.278 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:27:47.278 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:27:47.278 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:27:47.278 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:47.278 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.286+0000 7f9c1c257640 10 monclient: shutdown 2024-03-29T11:27:47.279 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.286+0000 7f9c1c257640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:47.279 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.286+0000 7f9c1c257640 1 -- 172.21.15.104:0/1269205616 >> v1:172.21.15.119:6789/0 conn(0x7f9c140b0770 legacy=0x7f9c14149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:47.279 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.286+0000 7f9c1c257640 1 -- 172.21.15.104:0/1269205616 shutdown_connections 2024-03-29T11:27:47.279 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.286+0000 7f9c1c257640 1 -- 172.21.15.104:0/1269205616 >> 172.21.15.104:0/1269205616 conn(0x7f9c1406b660 msgr2=0x7f9c1406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:47.279 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.286+0000 7f9c1c257640 1 -- 172.21.15.104:0/1269205616 shutdown_connections 2024-03-29T11:27:47.279 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.286+0000 7f9c1c257640 1 -- 172.21.15.104:0/1269205616 wait complete. 2024-03-29T11:27:47.283 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.287+0000 7f9c1c257640 10 monclient: build_initial_monmap 2024-03-29T11:27:47.283 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.287+0000 7f9c1c257640 10 monclient: monmap: 2024-03-29T11:27:47.283 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:27:47.283 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:47.283 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:27:47.283 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:27:47.283 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:47.284 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:47.284 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:27:47.284 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:27:47.284 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:27:47.284 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:47.284 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.287+0000 7f9c1c257640 1 librados: starting msgr at 2024-03-29T11:27:47.284 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.287+0000 7f9c1c257640 1 librados: starting objecter 2024-03-29T11:27:47.284 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.288+0000 7f9c1c257640 1 Processor -- start 2024-03-29T11:27:47.284 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.288+0000 7f9c1c257640 1 -- start start 2024-03-29T11:27:47.284 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.288+0000 7f9c1c257640 1 librados: setting wanted keys 2024-03-29T11:27:47.285 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.288+0000 7f9c1c257640 1 librados: calling monclient init 2024-03-29T11:27:47.285 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.288+0000 7f9c1c257640 10 monclient: init 2024-03-29T11:27:47.285 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.289+0000 7f9c1c257640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:47.285 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.289+0000 7f9c1c257640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:27:47.285 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.289+0000 7f9c1c257640 10 monclient(hunting): picked mon.noname-b con 0x7f9c140ac5a0 addr v1:172.21.15.119:6789/0 2024-03-29T11:27:47.285 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.289+0000 7f9c1c257640 10 monclient(hunting): picked mon.noname-a con 0x7f9c140b0770 addr v1:172.21.15.104:6789/0 2024-03-29T11:27:47.285 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.289+0000 7f9c1c257640 10 monclient(hunting): picked mon.noname-c con 0x7f9c1414a700 addr v1:172.21.15.104:6790/0 2024-03-29T11:27:47.285 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.289+0000 7f9c1c257640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c141dfc40 con 0x7f9c140b0770 2024-03-29T11:27:47.285 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.289+0000 7f9c1c257640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c141279f0 con 0x7f9c140ac5a0 2024-03-29T11:27:47.285 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.289+0000 7f9c1c257640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c141df910 con 0x7f9c1414a700 2024-03-29T11:27:47.285 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.289+0000 7f9c1c257640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:47.285 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.289+0000 7f9c1c257640 10 monclient(hunting): authenticate will time out at 1448.540216s 2024-03-29T11:27:47.285 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.289+0000 7f9c1a7f4640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f9c1414a700 0x7f9c141dbaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:51166/0 (socket says 172.21.15.104:51166) 2024-03-29T11:27:47.285 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.289+0000 7f9c1a7f4640 1 -- 172.21.15.104:0/2405586203 learned_addr learned my addr 172.21.15.104:0/2405586203 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:27:47.286 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.289+0000 7f9c0a7fc640 10 client.?.objecter ms_handle_connect 0x7f9c140ac5a0 2024-03-29T11:27:47.286 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.289+0000 7f9c0a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:47.286 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.289+0000 7f9c0a7fc640 10 client.?.objecter ms_handle_connect 0x7f9c1414a700 2024-03-29T11:27:47.286 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.289+0000 7f9c0a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:47.286 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.290+0000 7f9c0a7fc640 1 -- 172.21.15.104:0/2405586203 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2514823267 0 0) 0x7f9c141279f0 con 0x7f9c140ac5a0 2024-03-29T11:27:47.286 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.290+0000 7f9c0a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:27:47.286 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.290+0000 7f9c0a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:47.286 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.290+0000 7f9c0a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:47.286 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.290+0000 7f9c0a7fc640 10 monclient(hunting): my global_id is 4519 2024-03-29T11:27:47.286 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.290+0000 7f9c0a7fc640 1 -- 172.21.15.104:0/2405586203 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bf8004eb0 con 0x7f9c140ac5a0 2024-03-29T11:27:47.286 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.290+0000 7f9c0a7fc640 1 -- 172.21.15.104:0/2405586203 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2247943253 0 0) 0x7f9c141df910 con 0x7f9c1414a700 2024-03-29T11:27:47.286 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.290+0000 7f9c0a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:27:47.287 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.290+0000 7f9c0a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:47.287 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.290+0000 7f9c0a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:47.287 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.290+0000 7f9c0a7fc640 10 monclient(hunting): my global_id is 4553 2024-03-29T11:27:47.287 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.290+0000 7f9c0a7fc640 1 -- 172.21.15.104:0/2405586203 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c141279f0 con 0x7f9c1414a700 2024-03-29T11:27:47.287 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.290+0000 7f9c0a7fc640 10 client.?.objecter ms_handle_connect 0x7f9c140b0770 2024-03-29T11:27:47.287 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.290+0000 7f9c0a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:47.287 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.290+0000 7f9c0a7fc640 1 -- 172.21.15.104:0/2405586203 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2793090930 0 0) 0x7f9bf8004eb0 con 0x7f9c140ac5a0 2024-03-29T11:27:47.287 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.290+0000 7f9c0a7fc640 1 -- 172.21.15.104:0/2405586203 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c141df910 con 0x7f9c140ac5a0 2024-03-29T11:27:47.287 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.290+0000 7f9c0a7fc640 1 -- 172.21.15.104:0/2405586203 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f9c04001b60 con 0x7f9c140ac5a0 2024-03-29T11:27:47.287 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.290+0000 7f9c0a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:47.287 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.290+0000 7f9c0a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:47.287 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.290+0000 7f9c0a7fc640 10 monclient(hunting): dump: 2024-03-29T11:27:47.287 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:27:47.287 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:27:47.287 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:47.288 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:47.288 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:47.288 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:47.288 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:27:47.288 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:27:47.288 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:27:47.288 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:47.288 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.290+0000 7f9c0a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:47.289 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.291+0000 7f9c0a7fc640 1 -- 172.21.15.104:0/2405586203 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3786375272 0 0) 0x7f9c141279f0 con 0x7f9c1414a700 2024-03-29T11:27:47.289 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.291+0000 7f9c0a7fc640 1 -- 172.21.15.104:0/2405586203 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9bf8004eb0 con 0x7f9c1414a700 2024-03-29T11:27:47.289 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.291+0000 7f9c0a7fc640 1 -- 172.21.15.104:0/2405586203 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4103121200 0 0) 0x7f9c141df910 con 0x7f9c140ac5a0 2024-03-29T11:27:47.289 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.291+0000 7f9c0a7fc640 1 -- 172.21.15.104:0/2405586203 >> v1:172.21.15.104:6790/0 conn(0x7f9c1414a700 legacy=0x7f9c141dbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:47.289 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.291+0000 7f9c0a7fc640 1 -- 172.21.15.104:0/2405586203 >> v1:172.21.15.104:6789/0 conn(0x7f9c140b0770 legacy=0x7f9c14144390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:47.289 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.291+0000 7f9c0a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:27:47.289 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.291+0000 7f9c0a7fc640 1 monclient: found mon.b 2024-03-29T11:27:47.289 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.291+0000 7f9c0a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:47.289 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.291+0000 7f9c0a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:27:47.289 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.291+0000 7f9c0a7fc640 1 -- 172.21.15.104:0/2405586203 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c14070d10 con 0x7f9c140ac5a0 2024-03-29T11:27:47.289 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.291+0000 7f9c0a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:27:47.289 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.291+0000 7f9c0a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:27:47.289 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.291+0000 7f9c0a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:47.289 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.291+0000 7f9c1c257640 5 monclient: authenticate success, global_id 4519 2024-03-29T11:27:47.290 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.291+0000 7f9c1c257640 10 monclient: _renew_subs 2024-03-29T11:27:47.290 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.291+0000 7f9c1c257640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:27:47.290 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.291+0000 7f9c1c257640 1 -- 172.21.15.104:0/2405586203 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c14070f60 con 0x7f9c140ac5a0 2024-03-29T11:27:47.290 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.291+0000 7f9c1c257640 10 client.4519.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:47.290 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.291+0000 7f9c1c257640 10 monclient: _renew_subs 2024-03-29T11:27:47.290 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.291+0000 7f9c1c257640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:27:47.290 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.291+0000 7f9c1c257640 1 -- 172.21.15.104:0/2405586203 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c14144c80 con 0x7f9c140ac5a0 2024-03-29T11:27:47.290 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.291+0000 7f9c1c257640 1 librados: init done 2024-03-29T11:27:47.290 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.292+0000 7f9c0a7fc640 1 -- 172.21.15.104:0/2405586203 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c04004a90 con 0x7f9c140ac5a0 2024-03-29T11:27:47.290 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.292+0000 7f9c0a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:47.291 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.292+0000 7f9c0a7fc640 1 -- 172.21.15.104:0/2405586203 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f9c04004da0 con 0x7f9c140ac5a0 2024-03-29T11:27:47.291 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.292+0000 7f9c0a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:47.291 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.292+0000 7f9c0a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:47.291 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.292+0000 7f9c0a7fc640 10 monclient: dump: 2024-03-29T11:27:47.291 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:27:47.291 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:27:47.291 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:47.291 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:47.292 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:47.292 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:47.292 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:27:47.292 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:27:47.292 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:27:47.292 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:47.292 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.292+0000 7f9c0bfff640 4 set_mon_vals no callback set 2024-03-29T11:27:47.292 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.293+0000 7f9c1c257640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:47.292 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.294+0000 7f9c1c257640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:47.292 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.294+0000 7f9c1c257640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:27:47.292 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.294+0000 7f9c1c257640 1 -- 172.21.15.104:0/2405586203 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bfc005080 con 0x7f9c140ac5a0 2024-03-29T11:27:47.293 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.295+0000 7f9c0a7fc640 1 -- 172.21.15.104:0/2405586203 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f9c0401ad70 con 0x7f9c140ac5a0 2024-03-29T11:27:47.293 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.296+0000 7f9c197f2640 10 monclient: get_auth_request con 0x7f9bf806de80 auth_method 0 2024-03-29T11:27:47.293 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.296+0000 7f9c0a7fc640 1 -- 172.21.15.104:0/2405586203 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f9c04084ac0 con 0x7f9c140ac5a0 2024-03-29T11:27:47.293 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.296+0000 7f9c0a7fc640 10 client.4519.objecter ms_dispatch 0x7f9c14000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:47.294 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.296+0000 7f9c0a7fc640 3 client.4519.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:47.294 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.296+0000 7f9c0a7fc640 3 client.4519.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:47.294 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.296+0000 7f9c0a7fc640 20 client.4519.objecter dump_active .. 0 homeless 2024-03-29T11:27:47.294 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.296+0000 7f9c0a7fc640 10 client.4519.objecter ms_handle_connect 0x7f9bf806de80 2024-03-29T11:27:47.294 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.303+0000 7f9c0a7fc640 1 -- 172.21.15.104:0/2405586203 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9c040503a0 con 0x7f9c140ac5a0 2024-03-29T11:27:47.294 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.303+0000 7f9c0a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:47.294 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.303+0000 7f9c0a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:47.673 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.685+0000 7f9c1c257640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:47.673 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.685+0000 7f9c1c257640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:47.674 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.685+0000 7f9c1c257640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:27:47.674 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.685+0000 7f9c1c257640 1 -- 172.21.15.104:0/2405586203 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9bfc005330 con 0x7f9c140ac5a0 2024-03-29T11:27:47.675 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.686+0000 7f9c0a7fc640 1 -- 172.21.15.104:0/2405586203 <== mon.1 v1:172.21.15.119:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f9c040522d0 con 0x7f9c140ac5a0 2024-03-29T11:27:47.675 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.686+0000 7f9c0a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:47.675 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.686+0000 7f9c0a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:47.675 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:27:47.675 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:47.679 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.689+0000 7f9c1c257640 10 librados: watch_flush enter 2024-03-29T11:27:47.679 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.690+0000 7f9c1c257640 10 librados: watch_flush exit 2024-03-29T11:27:47.680 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.690+0000 7f9c1c257640 20 client.4519.objecter shutdown clearing up homeless session... 2024-03-29T11:27:47.680 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.690+0000 7f9c1c257640 10 client.4519.objecter successfully canceled tick 2024-03-29T11:27:47.680 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.690+0000 7f9c1c257640 1 -- 172.21.15.104:0/2405586203 >> v1:172.21.15.104:6816/3341126660 conn(0x7f9bf806de80 legacy=0x7f9bf8070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:47.680 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.690+0000 7f9c1c257640 10 monclient: shutdown 2024-03-29T11:27:47.680 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.690+0000 7f9c1c257640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:47.680 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.690+0000 7f9c1c257640 1 -- 172.21.15.104:0/2405586203 >> v1:172.21.15.119:6789/0 conn(0x7f9c140ac5a0 legacy=0x7f9c141df200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:47.680 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.690+0000 7f9c1c257640 1 -- 172.21.15.104:0/2405586203 shutdown_connections 2024-03-29T11:27:47.680 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.690+0000 7f9c1c257640 1 -- 172.21.15.104:0/2405586203 >> 172.21.15.104:0/2405586203 conn(0x7f9c1406b660 msgr2=0x7f9c140b3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:47.680 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.690+0000 7f9c1c257640 1 -- 172.21.15.104:0/2405586203 shutdown_connections 2024-03-29T11:27:47.680 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.690+0000 7f9c1c257640 1 -- 172.21.15.104:0/2405586203 wait complete. 2024-03-29T11:27:47.680 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:47.690+0000 7f9c1c257640 1 librados: shutdown 2024-03-29T11:27:47.693 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:54.501 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:54.650 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.661+0000 7f2801a8f640 10 monclient: get_monmap_and_config 2024-03-29T11:27:54.650 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.661+0000 7f2801a8f640 10 monclient: build_initial_monmap 2024-03-29T11:27:54.650 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.661+0000 7f2801a8f640 10 monclient: monmap: 2024-03-29T11:27:54.650 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:27:54.650 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:27:54.651 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:27:54.662600+0000 2024-03-29T11:27:54.651 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:27:54.662600+0000 2024-03-29T11:27:54.651 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:54.651 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:54.651 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:27:54.651 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:27:54.651 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:27:54.651 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:54.651 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.662+0000 7f2801a8f640 1 Processor -- start 2024-03-29T11:27:54.652 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.663+0000 7f2801a8f640 1 -- start start 2024-03-29T11:27:54.652 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.663+0000 7f2801a8f640 10 monclient: init 2024-03-29T11:27:54.653 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.664+0000 7f2801a8f640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:54.653 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.664+0000 7f2801a8f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:27:54.654 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.664+0000 7f2801a8f640 10 monclient(hunting): picked mon.noname-b con 0x7f27fc1457f0 addr v1:172.21.15.104:6790/0 2024-03-29T11:27:54.654 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.664+0000 7f2801a8f640 10 monclient(hunting): picked mon.noname-c con 0x7f27fc13f650 addr v1:172.21.15.119:6789/0 2024-03-29T11:27:54.654 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.664+0000 7f2801a8f640 10 monclient(hunting): picked mon.noname-a con 0x7f27fc0a9d70 addr v1:172.21.15.104:6789/0 2024-03-29T11:27:54.654 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.664+0000 7f2801a8f640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27fc0aaef0 con 0x7f27fc0a9d70 2024-03-29T11:27:54.654 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.664+0000 7f2801a8f640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27fc145bd0 con 0x7f27fc13f650 2024-03-29T11:27:54.654 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.664+0000 7f2801a8f640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27fc145eb0 con 0x7f27fc1457f0 2024-03-29T11:27:54.654 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.664+0000 7f2801a8f640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:54.654 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.664+0000 7f2801a8f640 10 monclient(hunting): authenticate will time out at 1455.915991s 2024-03-29T11:27:54.658 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.669+0000 7f27fb7fe640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f27fc1457f0 0x7f27fc13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:45529/0 (socket says 172.21.15.104:45529) 2024-03-29T11:27:54.659 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.669+0000 7f27fb7fe640 1 -- 172.21.15.104:0/4175646082 learned_addr learned my addr 172.21.15.104:0/4175646082 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:27:54.659 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.669+0000 7f27fa7fc640 1 -- 172.21.15.104:0/4175646082 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2052974634 0 0) 0x7f27fc145eb0 con 0x7f27fc1457f0 2024-03-29T11:27:54.659 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.670+0000 7f27fa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:27:54.659 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.670+0000 7f27fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:54.659 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.670+0000 7f27fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:54.659 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.670+0000 7f27fa7fc640 10 monclient(hunting): my global_id is 4556 2024-03-29T11:27:54.659 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.670+0000 7f27fa7fc640 1 -- 172.21.15.104:0/4175646082 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27e4003520 con 0x7f27fc1457f0 2024-03-29T11:27:54.659 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.670+0000 7f27fa7fc640 1 -- 172.21.15.104:0/4175646082 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1283039495 0 0) 0x7f27e4003520 con 0x7f27fc1457f0 2024-03-29T11:27:54.659 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.670+0000 7f27fa7fc640 1 -- 172.21.15.104:0/4175646082 >> v1:172.21.15.119:6789/0 conn(0x7f27fc13f650 legacy=0x7f27fc0b2620 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:27:54.659 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.670+0000 7f27fa7fc640 1 -- 172.21.15.104:0/4175646082 >> v1:172.21.15.104:6789/0 conn(0x7f27fc0a9d70 legacy=0x7f27fc14d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:54.662 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.671+0000 7f27fa7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:27:54.662 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.671+0000 7f27fa7fc640 1 monclient: found mon.noname-b 2024-03-29T11:27:54.662 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.671+0000 7f27fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:54.662 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.671+0000 7f27fa7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.104:6790/0 2024-03-29T11:27:54.662 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.671+0000 7f27fa7fc640 1 -- 172.21.15.104:0/4175646082 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27fc146340 con 0x7f27fc1457f0 2024-03-29T11:27:54.662 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.671+0000 7f27fa7fc640 10 monclient: _finish_auth 0 2024-03-29T11:27:54.662 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.671+0000 7f27fa7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:27:54.662 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.671+0000 7f27fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:54.662 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.671+0000 7f27fa7fc640 1 -- 172.21.15.104:0/4175646082 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f27f0001690 con 0x7f27fc1457f0 2024-03-29T11:27:54.663 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.671+0000 7f27fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:54.663 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.671+0000 7f27fa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:54.663 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.671+0000 7f27fa7fc640 10 monclient: dump: 2024-03-29T11:27:54.663 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:27:54.663 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:27:54.663 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:54.663 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:54.663 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:54.663 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:54.663 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:27:54.663 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:27:54.663 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:27:54.663 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:54.663 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.672+0000 7f2801a8f640 5 monclient: authenticate success, global_id 4556 2024-03-29T11:27:54.664 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.672+0000 7f2801a8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:54.664 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.672+0000 7f27fa7fc640 1 -- 172.21.15.104:0/4175646082 <== mon.2 v1:172.21.15.104:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27f0001860 con 0x7f27fc1457f0 2024-03-29T11:27:54.664 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.672+0000 7f27fa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:54.664 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.672+0000 7f27fa7fc640 1 -- 172.21.15.104:0/4175646082 <== mon.2 v1:172.21.15.104:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f27f0001cb0 con 0x7f27fc1457f0 2024-03-29T11:27:54.664 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.672+0000 7f27fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:54.664 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.672+0000 7f27fa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:54.664 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.672+0000 7f27fa7fc640 10 monclient: dump: 2024-03-29T11:27:54.664 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:27:54.664 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:27:54.664 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:54.664 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:54.664 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:54.664 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:54.664 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:27:54.664 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:27:54.665 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:27:54.665 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:54.665 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.673+0000 7f2801a8f640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:54.665 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.673+0000 7f2801a8f640 4 set_mon_vals no callback set 2024-03-29T11:27:54.666 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.674+0000 7f2801a8f640 10 monclient: shutdown 2024-03-29T11:27:54.666 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.674+0000 7f2801a8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:54.666 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.674+0000 7f2801a8f640 1 -- 172.21.15.104:0/4175646082 >> v1:172.21.15.104:6790/0 conn(0x7f27fc1457f0 legacy=0x7f27fc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:54.667 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.675+0000 7f2801a8f640 1 -- 172.21.15.104:0/4175646082 shutdown_connections 2024-03-29T11:27:54.667 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.675+0000 7f2801a8f640 1 -- 172.21.15.104:0/4175646082 >> 172.21.15.104:0/4175646082 conn(0x7f27fc066a00 msgr2=0x7f27fc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:54.667 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.675+0000 7f2801a8f640 1 -- 172.21.15.104:0/4175646082 shutdown_connections 2024-03-29T11:27:54.667 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.676+0000 7f2801a8f640 1 -- 172.21.15.104:0/4175646082 wait complete. 2024-03-29T11:27:54.667 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.677+0000 7f2801a8f640 10 monclient: build_initial_monmap 2024-03-29T11:27:54.667 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.677+0000 7f2801a8f640 10 monclient: monmap: 2024-03-29T11:27:54.667 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:27:54.667 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:54.667 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:27:54.667 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:27:54.667 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:54.667 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:54.667 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:27:54.668 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:27:54.668 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:27:54.668 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:54.668 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.677+0000 7f2801a8f640 1 librados: starting msgr at 2024-03-29T11:27:54.668 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.677+0000 7f2801a8f640 1 librados: starting objecter 2024-03-29T11:27:54.668 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.678+0000 7f2801a8f640 1 Processor -- start 2024-03-29T11:27:54.670 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.679+0000 7f2801a8f640 1 -- start start 2024-03-29T11:27:54.670 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.679+0000 7f2801a8f640 1 librados: setting wanted keys 2024-03-29T11:27:54.670 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.679+0000 7f2801a8f640 1 librados: calling monclient init 2024-03-29T11:27:54.670 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.679+0000 7f2801a8f640 10 monclient: init 2024-03-29T11:27:54.670 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.680+0000 7f2801a8f640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:54.670 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.680+0000 7f2801a8f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:27:54.670 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.680+0000 7f2801a8f640 10 monclient(hunting): picked mon.noname-c con 0x7f27fc0a9d70 addr v1:172.21.15.104:6790/0 2024-03-29T11:27:54.670 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.680+0000 7f2801a8f640 10 monclient(hunting): picked mon.noname-a con 0x7f27fc13f650 addr v1:172.21.15.104:6789/0 2024-03-29T11:27:54.670 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.680+0000 7f2801a8f640 10 monclient(hunting): picked mon.noname-b con 0x7f27fc1457f0 addr v1:172.21.15.119:6789/0 2024-03-29T11:27:54.670 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.681+0000 7f2801a8f640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27fc06f610 con 0x7f27fc13f650 2024-03-29T11:27:54.670 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.681+0000 7f2801a8f640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27fc071fa0 con 0x7f27fc1457f0 2024-03-29T11:27:54.670 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.681+0000 7f2801a8f640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27fc136e40 con 0x7f27fc0a9d70 2024-03-29T11:27:54.670 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.681+0000 7f2801a8f640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:54.671 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.681+0000 7f2801a8f640 10 monclient(hunting): authenticate will time out at 1455.932314s 2024-03-29T11:27:54.673 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.684+0000 7f27fbfff640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f27fc1457f0 0x7f27fc06e5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:34421/0 (socket says 172.21.15.104:34421) 2024-03-29T11:27:54.673 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.684+0000 7f27fbfff640 1 -- 172.21.15.104:0/1778026925 learned_addr learned my addr 172.21.15.104:0/1778026925 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:27:54.673 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.684+0000 7f27e3fff640 10 client.?.objecter ms_handle_connect 0x7f27fc1457f0 2024-03-29T11:27:54.673 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.684+0000 7f27e3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:54.674 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.685+0000 7f27e3fff640 10 client.?.objecter ms_handle_connect 0x7f27fc0a9d70 2024-03-29T11:27:54.674 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.685+0000 7f27e3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:54.674 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.685+0000 7f27e3fff640 1 -- 172.21.15.104:0/1778026925 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 191962410 0 0) 0x7f27fc136e40 con 0x7f27fc0a9d70 2024-03-29T11:27:54.674 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.685+0000 7f27e3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:27:54.674 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.685+0000 7f27e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:54.674 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.685+0000 7f27e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:54.674 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.685+0000 7f27e3fff640 10 monclient(hunting): my global_id is 4559 2024-03-29T11:27:54.674 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.685+0000 7f27e3fff640 1 -- 172.21.15.104:0/1778026925 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27e8004eb0 con 0x7f27fc0a9d70 2024-03-29T11:27:54.675 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.685+0000 7f27e3fff640 1 -- 172.21.15.104:0/1778026925 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2736600176 0 0) 0x7f27e8004eb0 con 0x7f27fc0a9d70 2024-03-29T11:27:54.675 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.685+0000 7f27e3fff640 1 -- 172.21.15.104:0/1778026925 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27fc136e40 con 0x7f27fc0a9d70 2024-03-29T11:27:54.675 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.685+0000 7f27e3fff640 1 -- 172.21.15.104:0/1778026925 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f27f0001b70 con 0x7f27fc0a9d70 2024-03-29T11:27:54.675 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.686+0000 7f27e3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:54.675 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.686+0000 7f27e3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:54.675 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.686+0000 7f27e3fff640 10 monclient(hunting): dump: 2024-03-29T11:27:54.675 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:27:54.675 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:27:54.675 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:54.675 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:54.675 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:54.675 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:54.675 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:27:54.675 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:27:54.676 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:27:54.676 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:54.676 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.686+0000 7f27e3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:54.677 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.689+0000 7f27e3fff640 1 -- 172.21.15.104:0/1778026925 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3886382033 0 0) 0x7f27fc071fa0 con 0x7f27fc1457f0 2024-03-29T11:27:54.678 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.689+0000 7f27e3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:27:54.678 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.689+0000 7f27e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:54.678 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.689+0000 7f27e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:54.678 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.689+0000 7f27e3fff640 10 monclient(hunting): my global_id is 4522 2024-03-29T11:27:54.678 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.689+0000 7f27e3fff640 1 -- 172.21.15.104:0/1778026925 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27e8004eb0 con 0x7f27fc1457f0 2024-03-29T11:27:54.678 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.689+0000 7f27e3fff640 1 -- 172.21.15.104:0/1778026925 <== mon.2 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4265735152 0 0) 0x7f27fc136e40 con 0x7f27fc0a9d70 2024-03-29T11:27:54.678 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.689+0000 7f27e3fff640 1 -- 172.21.15.104:0/1778026925 >> v1:172.21.15.119:6789/0 conn(0x7f27fc1457f0 legacy=0x7f27fc06e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:54.679 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.690+0000 7f27e3fff640 1 -- 172.21.15.104:0/1778026925 >> v1:172.21.15.104:6789/0 conn(0x7f27fc13f650 legacy=0x7f27fc06de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:54.679 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.690+0000 7f27e3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:27:54.679 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.690+0000 7f27e3fff640 1 monclient: found mon.c 2024-03-29T11:27:54.679 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.690+0000 7f27e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:54.679 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.690+0000 7f27e3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:27:54.679 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.690+0000 7f27e3fff640 1 -- 172.21.15.104:0/1778026925 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27fc06f0f0 con 0x7f27fc0a9d70 2024-03-29T11:27:54.679 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.690+0000 7f27e3fff640 10 monclient: _finish_auth 0 2024-03-29T11:27:54.679 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.690+0000 7f27e3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:27:54.679 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.690+0000 7f27e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:54.681 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.690+0000 7f2801a8f640 5 monclient: authenticate success, global_id 4559 2024-03-29T11:27:54.681 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.690+0000 7f2801a8f640 10 monclient: _renew_subs 2024-03-29T11:27:54.681 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.690+0000 7f2801a8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:27:54.682 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.690+0000 7f2801a8f640 1 -- 172.21.15.104:0/1778026925 --> v1:172.21.15.104:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27fc06ecb0 con 0x7f27fc0a9d70 2024-03-29T11:27:54.682 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.690+0000 7f2801a8f640 10 client.4559.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:54.682 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.690+0000 7f2801a8f640 10 monclient: _renew_subs 2024-03-29T11:27:54.682 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.690+0000 7f2801a8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:27:54.682 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.690+0000 7f2801a8f640 1 -- 172.21.15.104:0/1778026925 --> v1:172.21.15.104:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f27fc136720 con 0x7f27fc0a9d70 2024-03-29T11:27:54.682 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.690+0000 7f2801a8f640 1 librados: init done 2024-03-29T11:27:54.682 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.691+0000 7f27e3fff640 1 -- 172.21.15.104:0/1778026925 <== mon.2 v1:172.21.15.104:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27f0001cd0 con 0x7f27fc0a9d70 2024-03-29T11:27:54.682 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.691+0000 7f27e3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:54.682 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.691+0000 7f27f9ffb640 4 set_mon_vals no callback set 2024-03-29T11:27:54.682 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.692+0000 7f2801a8f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:54.682 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.692+0000 7f27e3fff640 1 -- 172.21.15.104:0/1778026925 <== mon.2 v1:172.21.15.104:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f27f0004c70 con 0x7f27fc0a9d70 2024-03-29T11:27:54.682 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.692+0000 7f27e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:54.682 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.692+0000 7f27e3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:54.682 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.692+0000 7f27e3fff640 10 monclient: dump: 2024-03-29T11:27:54.683 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:27:54.683 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:27:54.683 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:54.683 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:27:54.683 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:54.683 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:27:54.683 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:27:54.683 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:27:54.683 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:27:54.683 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:27:54.683 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.692+0000 7f2801a8f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:54.683 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.692+0000 7f2801a8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:27:54.683 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.692+0000 7f2801a8f640 1 -- 172.21.15.104:0/1778026925 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27dc005080 con 0x7f27fc0a9d70 2024-03-29T11:27:54.684 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.694+0000 7f27e3fff640 1 -- 172.21.15.104:0/1778026925 <== mon.2 v1:172.21.15.104:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f27f001ac40 con 0x7f27fc0a9d70 2024-03-29T11:27:54.684 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.694+0000 7f27e3fff640 1 -- 172.21.15.104:0/1778026925 <== mon.2 v1:172.21.15.104:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f27f0084ba0 con 0x7f27fc0a9d70 2024-03-29T11:27:54.684 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.694+0000 7f27e3fff640 10 client.4559.objecter ms_dispatch 0x7f27fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:54.685 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.694+0000 7f27e3fff640 3 client.4559.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:54.685 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.695+0000 7f27e3fff640 3 client.4559.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:54.685 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.695+0000 7f27e3fff640 20 client.4559.objecter dump_active .. 0 homeless 2024-03-29T11:27:54.685 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.695+0000 7f27faffd640 10 monclient: get_auth_request con 0x7f27e806dd00 auth_method 0 2024-03-29T11:27:54.685 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.696+0000 7f27e3fff640 10 client.4559.objecter ms_handle_connect 0x7f27e806dd00 2024-03-29T11:27:54.691 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.702+0000 7f27e3fff640 1 -- 172.21.15.104:0/1778026925 <== mon.2 v1:172.21.15.104:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f27f00502b0 con 0x7f27fc0a9d70 2024-03-29T11:27:54.691 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.702+0000 7f27e3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:54.691 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:54.702+0000 7f27e3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:55.036 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:55.047+0000 7f2801a8f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:55.037 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:55.047+0000 7f2801a8f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:55.037 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:55.048+0000 7f2801a8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:27:55.037 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:55.048+0000 7f2801a8f640 1 -- 172.21.15.104:0/1778026925 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f27dc005330 con 0x7f27fc0a9d70 2024-03-29T11:27:55.038 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:55.049+0000 7f27e3fff640 1 -- 172.21.15.104:0/1778026925 <== mon.2 v1:172.21.15.104:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f27f00523b0 con 0x7f27fc0a9d70 2024-03-29T11:27:55.038 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:55.049+0000 7f27e3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:55.038 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:55.049+0000 7f27e3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:55.038 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:27:55.038 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:55.041 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:55.052+0000 7f2801a8f640 10 librados: watch_flush enter 2024-03-29T11:27:55.041 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:55.052+0000 7f2801a8f640 10 librados: watch_flush exit 2024-03-29T11:27:55.041 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:55.052+0000 7f2801a8f640 20 client.4559.objecter shutdown clearing up homeless session... 2024-03-29T11:27:55.041 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:55.052+0000 7f2801a8f640 10 client.4559.objecter successfully canceled tick 2024-03-29T11:27:55.041 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:55.052+0000 7f2801a8f640 1 -- 172.21.15.104:0/1778026925 >> v1:172.21.15.104:6816/3341126660 conn(0x7f27e806dd00 legacy=0x7f27e80701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:55.042 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:55.052+0000 7f2801a8f640 10 monclient: shutdown 2024-03-29T11:27:55.042 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:55.052+0000 7f2801a8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:55.042 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:55.052+0000 7f2801a8f640 1 -- 172.21.15.104:0/1778026925 >> v1:172.21.15.104:6790/0 conn(0x7f27fc0a9d70 legacy=0x7f27fc137620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:55.042 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:55.053+0000 7f2801a8f640 1 -- 172.21.15.104:0/1778026925 shutdown_connections 2024-03-29T11:27:55.042 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:55.053+0000 7f2801a8f640 1 -- 172.21.15.104:0/1778026925 >> 172.21.15.104:0/1778026925 conn(0x7f27fc066a00 msgr2=0x7f27fc0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:55.043 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:55.053+0000 7f2801a8f640 1 -- 172.21.15.104:0/1778026925 shutdown_connections 2024-03-29T11:27:55.043 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:55.053+0000 7f2801a8f640 1 -- 172.21.15.104:0/1778026925 wait complete. 2024-03-29T11:27:55.043 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:27:55.053+0000 7f2801a8f640 1 librados: shutdown 2024-03-29T11:27:55.057 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:01.865 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:02.013 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.023+0000 7fd48ac7d640 10 monclient: get_monmap_and_config 2024-03-29T11:28:02.013 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.023+0000 7fd48ac7d640 10 monclient: build_initial_monmap 2024-03-29T11:28:02.013 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.023+0000 7fd48ac7d640 10 monclient: monmap: 2024-03-29T11:28:02.013 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:28:02.013 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:28:02.014 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:28:02.024312+0000 2024-03-29T11:28:02.014 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:28:02.024312+0000 2024-03-29T11:28:02.014 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:02.014 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:28:02.014 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:28:02.014 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:28:02.014 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:28:02.014 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:28:02.014 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.025+0000 7fd48ac7d640 1 Processor -- start 2024-03-29T11:28:02.015 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.025+0000 7fd48ac7d640 1 -- start start 2024-03-29T11:28:02.015 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.025+0000 7fd48ac7d640 10 monclient: init 2024-03-29T11:28:02.015 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.026+0000 7fd48ac7d640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:02.015 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.026+0000 7fd48ac7d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:28:02.015 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.026+0000 7fd48ac7d640 10 monclient(hunting): picked mon.noname-c con 0x7fd4840a9ac0 addr v1:172.21.15.119:6789/0 2024-03-29T11:28:02.015 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.026+0000 7fd48ac7d640 10 monclient(hunting): picked mon.noname-b con 0x7fd48413ee90 addr v1:172.21.15.104:6790/0 2024-03-29T11:28:02.016 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.026+0000 7fd48ac7d640 10 monclient(hunting): picked mon.noname-a con 0x7fd4840af900 addr v1:172.21.15.104:6789/0 2024-03-29T11:28:02.016 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.026+0000 7fd48ac7d640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4840afd10 con 0x7fd4840af900 2024-03-29T11:28:02.016 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.026+0000 7fd48ac7d640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4840af6f0 con 0x7fd4840a9ac0 2024-03-29T11:28:02.016 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.026+0000 7fd48ac7d640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4840aac80 con 0x7fd48413ee90 2024-03-29T11:28:02.016 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.026+0000 7fd48ac7d640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:02.016 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.026+0000 7fd48ac7d640 10 monclient(hunting): authenticate will time out at 1463.277823s 2024-03-29T11:28:02.019 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.030+0000 7fd483fff640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7fd48413ee90 0x7fd48413b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:46262/0 (socket says 172.21.15.104:46262) 2024-03-29T11:28:02.019 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.030+0000 7fd483fff640 1 -- 172.21.15.104:0/675554782 learned_addr learned my addr 172.21.15.104:0/675554782 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:28:02.019 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.030+0000 7fd4837fe640 1 -- 172.21.15.104:0/675554782 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1193458247 0 0) 0x7fd4840aac80 con 0x7fd48413ee90 2024-03-29T11:28:02.019 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.030+0000 7fd4837fe640 10 monclient(hunting): _negotiate 2024-03-29T11:28:02.019 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.030+0000 7fd4837fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:02.020 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.030+0000 7fd4837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:02.020 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.030+0000 7fd4837fe640 10 monclient(hunting): my global_id is 4562 2024-03-29T11:28:02.020 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.030+0000 7fd4837fe640 1 -- 172.21.15.104:0/675554782 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd46c003520 con 0x7fd48413ee90 2024-03-29T11:28:02.022 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.031+0000 7fd4837fe640 1 -- 172.21.15.104:0/675554782 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 883039237 0 0) 0x7fd46c003520 con 0x7fd48413ee90 2024-03-29T11:28:02.022 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.031+0000 7fd4837fe640 1 -- 172.21.15.104:0/675554782 >> v1:172.21.15.119:6789/0 conn(0x7fd4840a9ac0 legacy=0x7fd4840adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:02.022 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.031+0000 7fd4837fe640 1 -- 172.21.15.104:0/675554782 >> v1:172.21.15.104:6789/0 conn(0x7fd4840af900 legacy=0x7fd484149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:02.022 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.031+0000 7fd4837fe640 10 monclient: _finish_hunting 0 2024-03-29T11:28:02.022 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.031+0000 7fd4837fe640 1 monclient: found mon.noname-b 2024-03-29T11:28:02.023 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.031+0000 7fd4837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:02.023 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.031+0000 7fd4837fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.104:6790/0 2024-03-29T11:28:02.023 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.031+0000 7fd4837fe640 1 -- 172.21.15.104:0/675554782 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4840aaf60 con 0x7fd48413ee90 2024-03-29T11:28:02.023 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.031+0000 7fd4837fe640 10 monclient: _finish_auth 0 2024-03-29T11:28:02.023 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.031+0000 7fd4837fe640 10 monclient: _check_auth_tickets 2024-03-29T11:28:02.023 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.031+0000 7fd4837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:02.023 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.031+0000 7fd48ac7d640 5 monclient: authenticate success, global_id 4562 2024-03-29T11:28:02.023 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.031+0000 7fd48ac7d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:02.023 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.031+0000 7fd4837fe640 1 -- 172.21.15.104:0/675554782 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fd478001690 con 0x7fd48413ee90 2024-03-29T11:28:02.023 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.031+0000 7fd4837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:02.023 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.031+0000 7fd4837fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:02.023 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.031+0000 7fd4837fe640 10 monclient: dump: 2024-03-29T11:28:02.023 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:28:02.023 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:28:02.023 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:02.024 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:02.024 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:02.024 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:28:02.024 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:28:02.024 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:28:02.024 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:28:02.024 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:28:02.024 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.031+0000 7fd48ac7d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:02.024 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.031+0000 7fd4837fe640 1 -- 172.21.15.104:0/675554782 <== mon.2 v1:172.21.15.104:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd478001920 con 0x7fd48413ee90 2024-03-29T11:28:02.024 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.031+0000 7fd4837fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:02.024 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.031+0000 7fd48ac7d640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:02.025 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.031+0000 7fd48ac7d640 4 set_mon_vals no callback set 2024-03-29T11:28:02.025 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.032+0000 7fd48ac7d640 10 monclient: shutdown 2024-03-29T11:28:02.025 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.032+0000 7fd48ac7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:02.025 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.032+0000 7fd48ac7d640 1 -- 172.21.15.104:0/675554782 >> v1:172.21.15.104:6790/0 conn(0x7fd48413ee90 legacy=0x7fd48413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:02.025 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.033+0000 7fd48ac7d640 1 -- 172.21.15.104:0/675554782 shutdown_connections 2024-03-29T11:28:02.025 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.033+0000 7fd48ac7d640 1 -- 172.21.15.104:0/675554782 >> 172.21.15.104:0/675554782 conn(0x7fd4840b35e0 msgr2=0x7fd4840b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:02.025 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.033+0000 7fd48ac7d640 1 -- 172.21.15.104:0/675554782 shutdown_connections 2024-03-29T11:28:02.027 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.033+0000 7fd48ac7d640 1 -- 172.21.15.104:0/675554782 wait complete. 2024-03-29T11:28:02.027 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.034+0000 7fd48ac7d640 10 monclient: build_initial_monmap 2024-03-29T11:28:02.027 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.034+0000 7fd48ac7d640 10 monclient: monmap: 2024-03-29T11:28:02.027 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:28:02.027 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:02.028 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:28:02.028 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:28:02.028 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:02.028 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:28:02.028 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:28:02.028 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:28:02.028 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:28:02.028 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:28:02.028 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.035+0000 7fd48ac7d640 1 librados: starting msgr at 2024-03-29T11:28:02.028 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.035+0000 7fd48ac7d640 1 librados: starting objecter 2024-03-29T11:28:02.029 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.035+0000 7fd48ac7d640 1 Processor -- start 2024-03-29T11:28:02.029 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.035+0000 7fd48ac7d640 1 -- start start 2024-03-29T11:28:02.029 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.035+0000 7fd48ac7d640 1 librados: setting wanted keys 2024-03-29T11:28:02.029 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.035+0000 7fd48ac7d640 1 librados: calling monclient init 2024-03-29T11:28:02.029 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.035+0000 7fd48ac7d640 10 monclient: init 2024-03-29T11:28:02.029 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.036+0000 7fd48ac7d640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:02.029 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.036+0000 7fd48ac7d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:28:02.030 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.036+0000 7fd48ac7d640 10 monclient(hunting): picked mon.noname-c con 0x7fd4840a9ac0 addr v1:172.21.15.104:6790/0 2024-03-29T11:28:02.030 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.036+0000 7fd48ac7d640 10 monclient(hunting): picked mon.noname-a con 0x7fd4840af900 addr v1:172.21.15.104:6789/0 2024-03-29T11:28:02.030 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.036+0000 7fd48ac7d640 10 monclient(hunting): picked mon.noname-b con 0x7fd48413ee90 addr v1:172.21.15.119:6789/0 2024-03-29T11:28:02.030 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.036+0000 7fd48ac7d640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4841420d0 con 0x7fd4840af900 2024-03-29T11:28:02.030 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.036+0000 7fd48ac7d640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd484140f70 con 0x7fd48413ee90 2024-03-29T11:28:02.030 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.036+0000 7fd48ac7d640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4841424f0 con 0x7fd4840a9ac0 2024-03-29T11:28:02.030 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.036+0000 7fd48ac7d640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:02.030 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.036+0000 7fd48ac7d640 10 monclient(hunting): authenticate will time out at 1463.287991s 2024-03-29T11:28:02.030 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.037+0000 7fd488a19640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7fd4840a9ac0 0x7fd484130ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:48190/0 (socket says 172.21.15.104:48190) 2024-03-29T11:28:02.030 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.037+0000 7fd488a19640 1 -- 172.21.15.104:0/386275401 learned_addr learned my addr 172.21.15.104:0/386275401 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:28:02.030 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.037+0000 7fd4817fa640 10 client.?.objecter ms_handle_connect 0x7fd4840a9ac0 2024-03-29T11:28:02.030 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.037+0000 7fd4817fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:02.035 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.041+0000 7fd4817fa640 1 -- 172.21.15.104:0/386275401 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2568055604 0 0) 0x7fd4841424f0 con 0x7fd4840a9ac0 2024-03-29T11:28:02.035 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.041+0000 7fd4817fa640 10 monclient(hunting): _negotiate 2024-03-29T11:28:02.035 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.041+0000 7fd4817fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:02.035 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.041+0000 7fd4817fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:02.035 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.041+0000 7fd4817fa640 10 monclient(hunting): my global_id is 4565 2024-03-29T11:28:02.035 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.041+0000 7fd4817fa640 1 -- 172.21.15.104:0/386275401 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd470004eb0 con 0x7fd4840a9ac0 2024-03-29T11:28:02.035 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.041+0000 7fd4817fa640 1 -- 172.21.15.104:0/386275401 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1910232140 0 0) 0x7fd470004eb0 con 0x7fd4840a9ac0 2024-03-29T11:28:02.035 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.041+0000 7fd4817fa640 1 -- 172.21.15.104:0/386275401 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd4841424f0 con 0x7fd4840a9ac0 2024-03-29T11:28:02.035 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.041+0000 7fd4817fa640 1 -- 172.21.15.104:0/386275401 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fd47c001ce0 con 0x7fd4840a9ac0 2024-03-29T11:28:02.035 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.041+0000 7fd4817fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:02.036 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.041+0000 7fd4817fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:02.036 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.041+0000 7fd4817fa640 10 monclient(hunting): dump: 2024-03-29T11:28:02.036 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:28:02.036 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:28:02.036 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:02.036 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:02.036 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:02.036 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:28:02.036 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:28:02.036 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:28:02.036 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:28:02.036 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:28:02.036 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.041+0000 7fd4817fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:02.036 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.042+0000 7fd4817fa640 1 -- 172.21.15.104:0/386275401 <== mon.2 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4271113719 0 0) 0x7fd4841424f0 con 0x7fd4840a9ac0 2024-03-29T11:28:02.037 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.042+0000 7fd4817fa640 1 -- 172.21.15.104:0/386275401 >> v1:172.21.15.119:6789/0 conn(0x7fd48413ee90 legacy=0x7fd4841e6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:02.037 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.042+0000 7fd4817fa640 1 -- 172.21.15.104:0/386275401 >> v1:172.21.15.104:6789/0 conn(0x7fd4840af900 legacy=0x7fd4841ded50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:02.037 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.042+0000 7fd4817fa640 10 monclient: _finish_hunting 0 2024-03-29T11:28:02.037 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.042+0000 7fd4817fa640 1 monclient: found mon.c 2024-03-29T11:28:02.037 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.042+0000 7fd4817fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:02.037 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.042+0000 7fd4817fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:28:02.037 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.042+0000 7fd4817fa640 1 -- 172.21.15.104:0/386275401 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd484142680 con 0x7fd4840a9ac0 2024-03-29T11:28:02.037 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.042+0000 7fd4817fa640 10 monclient: _finish_auth 0 2024-03-29T11:28:02.037 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.042+0000 7fd4817fa640 10 monclient: _check_auth_tickets 2024-03-29T11:28:02.037 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.042+0000 7fd4817fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:02.037 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.042+0000 7fd4817fa640 1 -- 172.21.15.104:0/386275401 <== mon.2 v1:172.21.15.104:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd47c002000 con 0x7fd4840a9ac0 2024-03-29T11:28:02.037 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.042+0000 7fd4817fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:02.037 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.042+0000 7fd4817fa640 1 -- 172.21.15.104:0/386275401 <== mon.2 v1:172.21.15.104:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fd47c001ce0 con 0x7fd4840a9ac0 2024-03-29T11:28:02.037 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.042+0000 7fd4817fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:02.038 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.042+0000 7fd4817fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:02.038 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.042+0000 7fd4817fa640 10 monclient: dump: 2024-03-29T11:28:02.038 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:28:02.038 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:28:02.038 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:02.038 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:02.038 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:02.038 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:28:02.038 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:28:02.038 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:28:02.038 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:28:02.039 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:28:02.039 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.042+0000 7fd482ffd640 4 set_mon_vals no callback set 2024-03-29T11:28:02.039 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.043+0000 7fd48ac7d640 5 monclient: authenticate success, global_id 4565 2024-03-29T11:28:02.039 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.043+0000 7fd48ac7d640 10 monclient: _renew_subs 2024-03-29T11:28:02.039 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.043+0000 7fd48ac7d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:28:02.039 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.043+0000 7fd48ac7d640 1 -- 172.21.15.104:0/386275401 --> v1:172.21.15.104:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd48413fa00 con 0x7fd4840a9ac0 2024-03-29T11:28:02.039 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.043+0000 7fd48ac7d640 10 client.4565.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:02.039 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.043+0000 7fd48ac7d640 10 monclient: _renew_subs 2024-03-29T11:28:02.039 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.043+0000 7fd48ac7d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:28:02.039 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.043+0000 7fd48ac7d640 1 -- 172.21.15.104:0/386275401 --> v1:172.21.15.104:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4841317e0 con 0x7fd4840a9ac0 2024-03-29T11:28:02.040 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.043+0000 7fd48ac7d640 1 librados: init done 2024-03-29T11:28:02.040 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.044+0000 7fd48ac7d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:02.040 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.044+0000 7fd48ac7d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:02.040 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.044+0000 7fd48ac7d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:28:02.040 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.045+0000 7fd48ac7d640 1 -- 172.21.15.104:0/386275401 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd484075f10 con 0x7fd4840a9ac0 2024-03-29T11:28:02.040 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.045+0000 7fd4817fa640 1 -- 172.21.15.104:0/386275401 <== mon.2 v1:172.21.15.104:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7fd47c003c20 con 0x7fd4840a9ac0 2024-03-29T11:28:02.041 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.046+0000 7fd4817fa640 1 -- 172.21.15.104:0/386275401 <== mon.2 v1:172.21.15.104:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7fd47c084bb0 con 0x7fd4840a9ac0 2024-03-29T11:28:02.041 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.046+0000 7fd4817fa640 10 client.4565.objecter ms_dispatch 0x7fd484000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:02.041 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.046+0000 7fd4817fa640 3 client.4565.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:02.041 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.046+0000 7fd4817fa640 3 client.4565.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:02.041 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.046+0000 7fd4817fa640 20 client.4565.objecter dump_active .. 0 homeless 2024-03-29T11:28:02.041 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.046+0000 7fd483fff640 10 monclient: get_auth_request con 0x7fd47006ddd0 auth_method 0 2024-03-29T11:28:02.041 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.047+0000 7fd4817fa640 10 client.4565.objecter ms_handle_connect 0x7fd47006ddd0 2024-03-29T11:28:02.043 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.054+0000 7fd4817fa640 1 -- 172.21.15.104:0/386275401 <== mon.2 v1:172.21.15.104:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd47c050490 con 0x7fd4840a9ac0 2024-03-29T11:28:02.043 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.054+0000 7fd4817fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:02.043 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.054+0000 7fd4817fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:02.373 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.384+0000 7fd48ac7d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:02.374 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.384+0000 7fd48ac7d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:02.374 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.384+0000 7fd48ac7d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:28:02.374 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.384+0000 7fd48ac7d640 1 -- 172.21.15.104:0/386275401 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd48413fb90 con 0x7fd4840a9ac0 2024-03-29T11:28:02.375 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.386+0000 7fd4817fa640 1 -- 172.21.15.104:0/386275401 <== mon.2 v1:172.21.15.104:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7fd47c050490 con 0x7fd4840a9ac0 2024-03-29T11:28:02.375 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.386+0000 7fd4817fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:02.375 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.386+0000 7fd4817fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:02.375 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:28:02.376 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:02.382 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.389+0000 7fd48ac7d640 10 librados: watch_flush enter 2024-03-29T11:28:02.382 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.389+0000 7fd48ac7d640 10 librados: watch_flush exit 2024-03-29T11:28:02.382 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.389+0000 7fd48ac7d640 20 client.4565.objecter shutdown clearing up homeless session... 2024-03-29T11:28:02.382 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.389+0000 7fd48ac7d640 10 client.4565.objecter successfully canceled tick 2024-03-29T11:28:02.382 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.390+0000 7fd48ac7d640 1 -- 172.21.15.104:0/386275401 >> v1:172.21.15.104:6816/3341126660 conn(0x7fd47006ddd0 legacy=0x7fd470070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:02.382 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.390+0000 7fd48ac7d640 10 monclient: shutdown 2024-03-29T11:28:02.382 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.390+0000 7fd48ac7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:02.382 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.390+0000 7fd48ac7d640 1 -- 172.21.15.104:0/386275401 >> v1:172.21.15.104:6790/0 conn(0x7fd4840a9ac0 legacy=0x7fd484130ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:02.382 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.390+0000 7fd48ac7d640 1 -- 172.21.15.104:0/386275401 shutdown_connections 2024-03-29T11:28:02.382 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.390+0000 7fd48ac7d640 1 -- 172.21.15.104:0/386275401 >> 172.21.15.104:0/386275401 conn(0x7fd4840b35e0 msgr2=0x7fd4840ace60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:02.382 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.390+0000 7fd48ac7d640 1 -- 172.21.15.104:0/386275401 shutdown_connections 2024-03-29T11:28:02.382 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.390+0000 7fd48ac7d640 1 -- 172.21.15.104:0/386275401 wait complete. 2024-03-29T11:28:02.383 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:02.391+0000 7fd48ac7d640 1 librados: shutdown 2024-03-29T11:28:02.397 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:09.208 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:09.360 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.370+0000 7f697bda8640 10 monclient: get_monmap_and_config 2024-03-29T11:28:09.361 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.370+0000 7f697bda8640 10 monclient: build_initial_monmap 2024-03-29T11:28:09.361 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.370+0000 7f697bda8640 10 monclient: monmap: 2024-03-29T11:28:09.361 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:28:09.361 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:28:09.361 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:28:09.371845+0000 2024-03-29T11:28:09.361 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:28:09.371845+0000 2024-03-29T11:28:09.361 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:09.361 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:28:09.361 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:28:09.361 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:28:09.361 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:28:09.362 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:28:09.362 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.372+0000 7f697bda8640 1 Processor -- start 2024-03-29T11:28:09.362 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.372+0000 7f697bda8640 1 -- start start 2024-03-29T11:28:09.362 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.372+0000 7f697bda8640 10 monclient: init 2024-03-29T11:28:09.363 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.374+0000 7f697bda8640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:09.363 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.374+0000 7f697bda8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:28:09.363 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.374+0000 7f697bda8640 10 monclient(hunting): picked mon.noname-a con 0x7f69740b02e0 addr v1:172.21.15.104:6789/0 2024-03-29T11:28:09.366 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.375+0000 7f697bda8640 10 monclient(hunting): picked mon.noname-c con 0x7f69740aeb90 addr v1:172.21.15.119:6789/0 2024-03-29T11:28:09.367 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.375+0000 7f697bda8640 10 monclient(hunting): picked mon.noname-b con 0x7f69740aeff0 addr v1:172.21.15.104:6790/0 2024-03-29T11:28:09.367 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.375+0000 7f697a345640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f69740aeff0 0x7f697414da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:46970/0 (socket says 172.21.15.104:46970) 2024-03-29T11:28:09.367 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.375+0000 7f697a345640 1 -- 172.21.15.104:0/580420072 learned_addr learned my addr 172.21.15.104:0/580420072 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:28:09.367 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.375+0000 7f697bda8640 1 -- 172.21.15.104:0/580420072 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f697413fb10 con 0x7f69740b02e0 2024-03-29T11:28:09.367 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.375+0000 7f697bda8640 1 -- 172.21.15.104:0/580420072 --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69740ae280 con 0x7f69740aeb90 2024-03-29T11:28:09.367 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.375+0000 7f697bda8640 1 -- 172.21.15.104:0/580420072 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69741421e0 con 0x7f69740aeff0 2024-03-29T11:28:09.367 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.375+0000 7f697bda8640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:09.367 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.375+0000 7f697bda8640 10 monclient(hunting): authenticate will time out at 1470.626579s 2024-03-29T11:28:09.367 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.376+0000 7f6978b42640 1 -- 172.21.15.104:0/580420072 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3678772877 0 0) 0x7f69741421e0 con 0x7f69740aeff0 2024-03-29T11:28:09.367 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.376+0000 7f6978b42640 10 monclient(hunting): _negotiate 2024-03-29T11:28:09.367 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.376+0000 7f6978b42640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:09.367 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.376+0000 7f6978b42640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:09.368 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.376+0000 7f6978b42640 10 monclient(hunting): my global_id is 4568 2024-03-29T11:28:09.368 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.376+0000 7f6978b42640 1 -- 172.21.15.104:0/580420072 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f695c003520 con 0x7f69740aeff0 2024-03-29T11:28:09.368 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.377+0000 7f6978b42640 1 -- 172.21.15.104:0/580420072 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3911756058 0 0) 0x7f695c003520 con 0x7f69740aeff0 2024-03-29T11:28:09.368 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.377+0000 7f6978b42640 1 -- 172.21.15.104:0/580420072 >> v1:172.21.15.119:6789/0 conn(0x7f69740aeb90 legacy=0x7f6974142700 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:09.368 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.379+0000 7f6978b42640 1 -- 172.21.15.104:0/580420072 >> v1:172.21.15.104:6789/0 conn(0x7f69740b02e0 legacy=0x7f697413f400 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:09.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.379+0000 7f6978b42640 10 monclient: _finish_hunting 0 2024-03-29T11:28:09.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.379+0000 7f6978b42640 1 monclient: found mon.noname-b 2024-03-29T11:28:09.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.379+0000 7f6978b42640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:09.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.379+0000 7f6978b42640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.104:6790/0 2024-03-29T11:28:09.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.379+0000 7f6978b42640 1 -- 172.21.15.104:0/580420072 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6974145660 con 0x7f69740aeff0 2024-03-29T11:28:09.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.379+0000 7f6978b42640 10 monclient: _finish_auth 0 2024-03-29T11:28:09.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.379+0000 7f6978b42640 10 monclient: _check_auth_tickets 2024-03-29T11:28:09.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.379+0000 7f6978b42640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:09.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.379+0000 7f6978b42640 1 -- 172.21.15.104:0/580420072 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f6964001a10 con 0x7f69740aeff0 2024-03-29T11:28:09.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.379+0000 7f6978b42640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:09.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.379+0000 7f6978b42640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:09.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.379+0000 7f6978b42640 10 monclient: dump: 2024-03-29T11:28:09.369 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:28:09.370 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:28:09.370 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:09.370 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:09.370 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:09.370 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:28:09.370 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:28:09.370 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:28:09.370 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:28:09.370 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:28:09.371 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.381+0000 7f697bda8640 5 monclient: authenticate success, global_id 4568 2024-03-29T11:28:09.371 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.381+0000 7f697bda8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:09.371 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.381+0000 7f6978b42640 1 -- 172.21.15.104:0/580420072 <== mon.2 v1:172.21.15.104:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6964001be0 con 0x7f69740aeff0 2024-03-29T11:28:09.371 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.381+0000 7f6978b42640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:09.371 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.381+0000 7f6978b42640 1 -- 172.21.15.104:0/580420072 <== mon.2 v1:172.21.15.104:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f6964002030 con 0x7f69740aeff0 2024-03-29T11:28:09.371 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.381+0000 7f6978b42640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:09.371 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.381+0000 7f6978b42640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:09.371 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.381+0000 7f6978b42640 10 monclient: dump: 2024-03-29T11:28:09.372 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:28:09.372 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:28:09.372 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:09.372 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:09.372 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:09.372 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:28:09.372 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:28:09.372 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:28:09.372 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:28:09.372 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:28:09.372 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.381+0000 7f697bda8640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:09.372 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.381+0000 7f697bda8640 4 set_mon_vals no callback set 2024-03-29T11:28:09.373 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.382+0000 7f697bda8640 10 monclient: shutdown 2024-03-29T11:28:09.373 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.382+0000 7f697bda8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:09.373 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.382+0000 7f697bda8640 1 -- 172.21.15.104:0/580420072 >> v1:172.21.15.104:6790/0 conn(0x7f69740aeff0 legacy=0x7f697414da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:09.373 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.382+0000 7f697bda8640 1 -- 172.21.15.104:0/580420072 shutdown_connections 2024-03-29T11:28:09.373 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.382+0000 7f697bda8640 1 -- 172.21.15.104:0/580420072 >> 172.21.15.104:0/580420072 conn(0x7f697406b660 msgr2=0x7f697406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:09.373 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.383+0000 7f697bda8640 1 -- 172.21.15.104:0/580420072 shutdown_connections 2024-03-29T11:28:09.375 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.385+0000 7f697bda8640 1 -- 172.21.15.104:0/580420072 wait complete. 2024-03-29T11:28:09.376 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.386+0000 7f697bda8640 10 monclient: build_initial_monmap 2024-03-29T11:28:09.376 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.386+0000 7f697bda8640 10 monclient: monmap: 2024-03-29T11:28:09.376 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:28:09.377 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:09.377 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:28:09.377 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:28:09.377 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:09.377 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:28:09.377 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:28:09.377 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:28:09.377 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:28:09.377 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:28:09.377 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.387+0000 7f697bda8640 1 librados: starting msgr at 2024-03-29T11:28:09.377 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.387+0000 7f697bda8640 1 librados: starting objecter 2024-03-29T11:28:09.377 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.387+0000 7f697bda8640 1 Processor -- start 2024-03-29T11:28:09.385 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.387+0000 7f697bda8640 1 -- start start 2024-03-29T11:28:09.385 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.387+0000 7f697bda8640 1 librados: setting wanted keys 2024-03-29T11:28:09.386 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.387+0000 7f697bda8640 1 librados: calling monclient init 2024-03-29T11:28:09.386 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.387+0000 7f697bda8640 10 monclient: init 2024-03-29T11:28:09.386 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.388+0000 7f697bda8640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:09.386 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.388+0000 7f697bda8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:28:09.386 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.388+0000 7f697bda8640 10 monclient(hunting): picked mon.noname-a con 0x7f69740aeb90 addr v1:172.21.15.104:6789/0 2024-03-29T11:28:09.386 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.388+0000 7f697bda8640 10 monclient(hunting): picked mon.noname-c con 0x7f69740aeff0 addr v1:172.21.15.104:6790/0 2024-03-29T11:28:09.387 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.388+0000 7f6979343640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f69740aeff0 0x7f69740ab720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:54116/0 (socket says 172.21.15.104:54116) 2024-03-29T11:28:09.387 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.388+0000 7f6979343640 1 -- 172.21.15.104:0/4152266737 learned_addr learned my addr 172.21.15.104:0/4152266737 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:28:09.387 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.388+0000 7f697bda8640 10 monclient(hunting): picked mon.noname-b con 0x7f69740b02e0 addr v1:172.21.15.119:6789/0 2024-03-29T11:28:09.387 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.388+0000 7f697bda8640 1 -- 172.21.15.104:0/4152266737 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f697406fe40 con 0x7f69740aeb90 2024-03-29T11:28:09.387 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.388+0000 7f697bda8640 1 -- 172.21.15.104:0/4152266737 --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f697406f310 con 0x7f69740b02e0 2024-03-29T11:28:09.387 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.388+0000 7f697bda8640 1 -- 172.21.15.104:0/4152266737 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69740700b0 con 0x7f69740aeff0 2024-03-29T11:28:09.387 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.388+0000 7f697bda8640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:09.387 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.388+0000 7f697bda8640 10 monclient(hunting): authenticate will time out at 1470.639969s 2024-03-29T11:28:09.387 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.390+0000 7f69727fc640 10 client.?.objecter ms_handle_connect 0x7f69740aeff0 2024-03-29T11:28:09.388 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.390+0000 7f69727fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:09.388 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.390+0000 7f69727fc640 1 -- 172.21.15.104:0/4152266737 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2820390507 0 0) 0x7f69740700b0 con 0x7f69740aeff0 2024-03-29T11:28:09.388 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.391+0000 7f69727fc640 10 monclient(hunting): _negotiate 2024-03-29T11:28:09.388 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.391+0000 7f69727fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:09.388 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.391+0000 7f69727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:09.388 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.391+0000 7f69727fc640 10 monclient(hunting): my global_id is 4571 2024-03-29T11:28:09.388 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.391+0000 7f69727fc640 1 -- 172.21.15.104:0/4152266737 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6960004eb0 con 0x7f69740aeff0 2024-03-29T11:28:09.388 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.391+0000 7f69727fc640 1 -- 172.21.15.104:0/4152266737 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1733262566 0 0) 0x7f6960004eb0 con 0x7f69740aeff0 2024-03-29T11:28:09.388 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.391+0000 7f69727fc640 1 -- 172.21.15.104:0/4152266737 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f69740700b0 con 0x7f69740aeff0 2024-03-29T11:28:09.389 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.391+0000 7f69727fc640 1 -- 172.21.15.104:0/4152266737 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f6968001960 con 0x7f69740aeff0 2024-03-29T11:28:09.389 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.391+0000 7f69727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:09.389 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.391+0000 7f69727fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:09.389 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.391+0000 7f69727fc640 10 monclient(hunting): dump: 2024-03-29T11:28:09.389 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:28:09.389 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:28:09.389 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:09.389 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:09.389 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:09.389 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:28:09.389 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:28:09.390 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:28:09.390 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:28:09.390 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:28:09.390 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.391+0000 7f69727fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:09.390 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.392+0000 7f69727fc640 1 -- 172.21.15.104:0/4152266737 <== mon.2 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1407873153 0 0) 0x7f69740700b0 con 0x7f69740aeff0 2024-03-29T11:28:09.390 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.392+0000 7f69727fc640 1 -- 172.21.15.104:0/4152266737 >> v1:172.21.15.119:6789/0 conn(0x7f69740b02e0 legacy=0x7f69740abe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:09.390 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.392+0000 7f69727fc640 1 -- 172.21.15.104:0/4152266737 >> v1:172.21.15.104:6789/0 conn(0x7f69740aeb90 legacy=0x7f697406ea60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:09.390 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.395+0000 7f69727fc640 10 monclient: _finish_hunting 0 2024-03-29T11:28:09.390 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.395+0000 7f69727fc640 1 monclient: found mon.c 2024-03-29T11:28:09.390 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.395+0000 7f69727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:09.390 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.395+0000 7f69727fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:28:09.390 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.395+0000 7f69727fc640 1 -- 172.21.15.104:0/4152266737 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6974070390 con 0x7f69740aeff0 2024-03-29T11:28:09.390 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.395+0000 7f69727fc640 10 monclient: _finish_auth 0 2024-03-29T11:28:09.390 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.395+0000 7f69727fc640 10 monclient: _check_auth_tickets 2024-03-29T11:28:09.391 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.395+0000 7f69727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:09.391 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.395+0000 7f697bda8640 5 monclient: authenticate success, global_id 4571 2024-03-29T11:28:09.391 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.395+0000 7f697bda8640 10 monclient: _renew_subs 2024-03-29T11:28:09.391 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.395+0000 7f697bda8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:28:09.391 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.395+0000 7f697bda8640 1 -- 172.21.15.104:0/4152266737 --> v1:172.21.15.104:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6974071aa0 con 0x7f69740aeff0 2024-03-29T11:28:09.391 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.395+0000 7f697bda8640 10 client.4571.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:09.391 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.395+0000 7f697bda8640 10 monclient: _renew_subs 2024-03-29T11:28:09.391 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.395+0000 7f697bda8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:28:09.391 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.395+0000 7f697bda8640 1 -- 172.21.15.104:0/4152266737 --> v1:172.21.15.104:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f69741e76b0 con 0x7f69740aeff0 2024-03-29T11:28:09.391 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.395+0000 7f697bda8640 1 librados: init done 2024-03-29T11:28:09.392 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.398+0000 7f69727fc640 1 -- 172.21.15.104:0/4152266737 <== mon.2 v1:172.21.15.104:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6968001d40 con 0x7f69740aeff0 2024-03-29T11:28:09.393 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.398+0000 7f69727fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:09.393 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.398+0000 7f69727fc640 1 -- 172.21.15.104:0/4152266737 <== mon.2 v1:172.21.15.104:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f6968001bc0 con 0x7f69740aeff0 2024-03-29T11:28:09.393 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.398+0000 7f69727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:09.393 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.398+0000 7f6973fff640 4 set_mon_vals no callback set 2024-03-29T11:28:09.393 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.398+0000 7f69727fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:09.393 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.398+0000 7f69727fc640 10 monclient: dump: 2024-03-29T11:28:09.393 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:28:09.393 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:28:09.393 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:09.393 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:09.393 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:09.393 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:28:09.393 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:28:09.393 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:28:09.394 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:28:09.394 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:28:09.394 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.398+0000 7f697bda8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:09.394 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.398+0000 7f697bda8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:09.394 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.398+0000 7f697bda8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:28:09.394 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.399+0000 7f697bda8640 1 -- 172.21.15.104:0/4152266737 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6954005080 con 0x7f69740aeff0 2024-03-29T11:28:09.394 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.399+0000 7f69727fc640 1 -- 172.21.15.104:0/4152266737 <== mon.2 v1:172.21.15.104:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f6968003b00 con 0x7f69740aeff0 2024-03-29T11:28:09.394 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.399+0000 7f69727fc640 1 -- 172.21.15.104:0/4152266737 <== mon.2 v1:172.21.15.104:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f6968084b40 con 0x7f69740aeff0 2024-03-29T11:28:09.394 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.399+0000 7f69727fc640 10 client.4571.objecter ms_dispatch 0x7f6974000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:09.394 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.399+0000 7f69727fc640 3 client.4571.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:09.394 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.399+0000 7f69727fc640 3 client.4571.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:09.394 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.399+0000 7f69727fc640 20 client.4571.objecter dump_active .. 0 homeless 2024-03-29T11:28:09.394 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.400+0000 7f6979b44640 10 monclient: get_auth_request con 0x7f696006dd00 auth_method 0 2024-03-29T11:28:09.394 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.400+0000 7f69727fc640 10 client.4571.objecter ms_handle_connect 0x7f696006dd00 2024-03-29T11:28:09.396 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.407+0000 7f69727fc640 1 -- 172.21.15.104:0/4152266737 <== mon.2 v1:172.21.15.104:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6968050420 con 0x7f69740aeff0 2024-03-29T11:28:09.396 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.407+0000 7f69727fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:09.396 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.407+0000 7f69727fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:09.723 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.733+0000 7f697bda8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:09.723 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.733+0000 7f697bda8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:09.723 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.733+0000 7f697bda8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:28:09.723 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.733+0000 7f697bda8640 1 -- 172.21.15.104:0/4152266737 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6954005330 con 0x7f69740aeff0 2024-03-29T11:28:09.724 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:28:09.724 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:09.725 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.735+0000 7f69727fc640 1 -- 172.21.15.104:0/4152266737 <== mon.2 v1:172.21.15.104:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f6968052350 con 0x7f69740aeff0 2024-03-29T11:28:09.725 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.735+0000 7f69727fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:09.725 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.735+0000 7f69727fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:09.727 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.738+0000 7f697bda8640 10 librados: watch_flush enter 2024-03-29T11:28:09.727 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.738+0000 7f697bda8640 10 librados: watch_flush exit 2024-03-29T11:28:09.728 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.738+0000 7f697bda8640 20 client.4571.objecter shutdown clearing up homeless session... 2024-03-29T11:28:09.728 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.738+0000 7f697bda8640 10 client.4571.objecter successfully canceled tick 2024-03-29T11:28:09.729 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.738+0000 7f697bda8640 1 -- 172.21.15.104:0/4152266737 >> v1:172.21.15.104:6816/3341126660 conn(0x7f696006dd00 legacy=0x7f69600701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:09.729 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.739+0000 7f697bda8640 10 monclient: shutdown 2024-03-29T11:28:09.729 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.739+0000 7f697bda8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:09.729 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.739+0000 7f697bda8640 1 -- 172.21.15.104:0/4152266737 >> v1:172.21.15.104:6790/0 conn(0x7f69740aeff0 legacy=0x7f69740ab720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:09.729 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.739+0000 7f697bda8640 1 -- 172.21.15.104:0/4152266737 shutdown_connections 2024-03-29T11:28:09.729 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.739+0000 7f697bda8640 1 -- 172.21.15.104:0/4152266737 >> 172.21.15.104:0/4152266737 conn(0x7f697406b660 msgr2=0x7f69740a9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:09.729 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.739+0000 7f697bda8640 1 -- 172.21.15.104:0/4152266737 shutdown_connections 2024-03-29T11:28:09.729 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.739+0000 7f697bda8640 1 -- 172.21.15.104:0/4152266737 wait complete. 2024-03-29T11:28:09.729 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:09.739+0000 7f697bda8640 1 librados: shutdown 2024-03-29T11:28:09.745 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:16.553 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:16.687 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.698+0000 7fa1f4c06640 10 monclient: get_monmap_and_config 2024-03-29T11:28:16.687 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.698+0000 7fa1f4c06640 10 monclient: build_initial_monmap 2024-03-29T11:28:16.688 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.698+0000 7fa1f4c06640 10 monclient: monmap: 2024-03-29T11:28:16.688 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:28:16.688 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:28:16.688 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:28:16.699172+0000 2024-03-29T11:28:16.688 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:28:16.699172+0000 2024-03-29T11:28:16.688 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:16.688 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:28:16.688 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:28:16.688 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:28:16.688 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:28:16.688 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:28:16.689 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.699+0000 7fa1f4c06640 1 Processor -- start 2024-03-29T11:28:16.689 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.699+0000 7fa1f4c06640 1 -- start start 2024-03-29T11:28:16.689 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.699+0000 7fa1f4c06640 10 monclient: init 2024-03-29T11:28:16.690 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.700+0000 7fa1f4c06640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:16.691 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.700+0000 7fa1f4c06640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:28:16.691 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.700+0000 7fa1f4c06640 10 monclient(hunting): picked mon.noname-b con 0x7fa1f01457f0 addr v1:172.21.15.104:6790/0 2024-03-29T11:28:16.691 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.700+0000 7fa1f4c06640 10 monclient(hunting): picked mon.noname-a con 0x7fa1f013f650 addr v1:172.21.15.104:6789/0 2024-03-29T11:28:16.691 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.700+0000 7fa1f4c06640 10 monclient(hunting): picked mon.noname-c con 0x7fa1f00a9d70 addr v1:172.21.15.119:6789/0 2024-03-29T11:28:16.691 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.700+0000 7fa1f4c06640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f00aaef0 con 0x7fa1f013f650 2024-03-29T11:28:16.691 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.700+0000 7fa1f4c06640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f0145bd0 con 0x7fa1f00a9d70 2024-03-29T11:28:16.691 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.700+0000 7fa1f4c06640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f0145eb0 con 0x7fa1f01457f0 2024-03-29T11:28:16.692 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.700+0000 7fa1f4c06640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:16.692 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.700+0000 7fa1f4c06640 10 monclient(hunting): authenticate will time out at 1477.952000s 2024-03-29T11:28:16.692 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.700+0000 7fa1ee59c640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7fa1f01457f0 0x7fa1f013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:59102/0 (socket says 172.21.15.104:59102) 2024-03-29T11:28:16.692 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.701+0000 7fa1ee59c640 1 -- 172.21.15.104:0/2473091276 learned_addr learned my addr 172.21.15.104:0/2473091276 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:28:16.692 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.701+0000 7fa1ed59a640 1 -- 172.21.15.104:0/2473091276 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 100579714 0 0) 0x7fa1f0145eb0 con 0x7fa1f01457f0 2024-03-29T11:28:16.692 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.701+0000 7fa1ed59a640 10 monclient(hunting): _negotiate 2024-03-29T11:28:16.692 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.701+0000 7fa1ed59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:16.692 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.701+0000 7fa1ed59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:16.692 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.701+0000 7fa1ed59a640 10 monclient(hunting): my global_id is 4574 2024-03-29T11:28:16.698 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.701+0000 7fa1ed59a640 1 -- 172.21.15.104:0/2473091276 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1d8003520 con 0x7fa1f01457f0 2024-03-29T11:28:16.699 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.701+0000 7fa1ed59a640 1 -- 172.21.15.104:0/2473091276 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2662380750 0 0) 0x7fa1d8003520 con 0x7fa1f01457f0 2024-03-29T11:28:16.699 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.701+0000 7fa1ed59a640 1 -- 172.21.15.104:0/2473091276 >> v1:172.21.15.119:6789/0 conn(0x7fa1f00a9d70 legacy=0x7fa1f014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:16.699 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.701+0000 7fa1ed59a640 1 -- 172.21.15.104:0/2473091276 >> v1:172.21.15.104:6789/0 conn(0x7fa1f013f650 legacy=0x7fa1f00b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:16.708 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.703+0000 7fa1ed59a640 10 monclient: _finish_hunting 0 2024-03-29T11:28:16.708 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.703+0000 7fa1ed59a640 1 monclient: found mon.noname-b 2024-03-29T11:28:16.708 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.703+0000 7fa1ed59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:16.709 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.703+0000 7fa1ed59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.104:6790/0 2024-03-29T11:28:16.709 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.703+0000 7fa1ed59a640 1 -- 172.21.15.104:0/2473091276 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f0146340 con 0x7fa1f01457f0 2024-03-29T11:28:16.709 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.703+0000 7fa1ed59a640 10 monclient: _finish_auth 0 2024-03-29T11:28:16.709 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.703+0000 7fa1ed59a640 10 monclient: _check_auth_tickets 2024-03-29T11:28:16.709 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.703+0000 7fa1ed59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:16.709 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.703+0000 7fa1ed59a640 1 -- 172.21.15.104:0/2473091276 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fa1e4001690 con 0x7fa1f01457f0 2024-03-29T11:28:16.709 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.703+0000 7fa1ed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:16.709 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.703+0000 7fa1ed59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:16.710 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.703+0000 7fa1ed59a640 10 monclient: dump: 2024-03-29T11:28:16.710 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:28:16.710 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:28:16.710 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:16.710 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:16.710 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:16.710 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:28:16.710 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:28:16.710 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:28:16.710 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:28:16.710 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:28:16.711 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.703+0000 7fa1f4c06640 5 monclient: authenticate success, global_id 4574 2024-03-29T11:28:16.711 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.703+0000 7fa1f4c06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:16.711 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.704+0000 7fa1ed59a640 1 -- 172.21.15.104:0/2473091276 <== mon.2 v1:172.21.15.104:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa1e4001860 con 0x7fa1f01457f0 2024-03-29T11:28:16.711 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.704+0000 7fa1ed59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:16.711 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.704+0000 7fa1f4c06640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:16.711 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.704+0000 7fa1f4c06640 4 set_mon_vals no callback set 2024-03-29T11:28:16.711 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.705+0000 7fa1ed59a640 1 -- 172.21.15.104:0/2473091276 <== mon.2 v1:172.21.15.104:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fa1e4001cb0 con 0x7fa1f01457f0 2024-03-29T11:28:16.711 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.705+0000 7fa1ed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:16.711 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.705+0000 7fa1ed59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:16.711 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.705+0000 7fa1ed59a640 10 monclient: dump: 2024-03-29T11:28:16.711 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:28:16.711 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:28:16.712 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:16.712 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:16.712 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:16.712 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:28:16.712 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:28:16.712 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:28:16.712 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:28:16.712 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:28:16.712 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.705+0000 7fa1f4c06640 10 monclient: shutdown 2024-03-29T11:28:16.713 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.705+0000 7fa1f4c06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:16.713 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.705+0000 7fa1f4c06640 1 -- 172.21.15.104:0/2473091276 >> v1:172.21.15.104:6790/0 conn(0x7fa1f01457f0 legacy=0x7fa1f013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:16.713 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.705+0000 7fa1f4c06640 1 -- 172.21.15.104:0/2473091276 shutdown_connections 2024-03-29T11:28:16.713 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.705+0000 7fa1f4c06640 1 -- 172.21.15.104:0/2473091276 >> 172.21.15.104:0/2473091276 conn(0x7fa1f0066a00 msgr2=0x7fa1f0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:16.713 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.705+0000 7fa1f4c06640 1 -- 172.21.15.104:0/2473091276 shutdown_connections 2024-03-29T11:28:16.713 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.705+0000 7fa1f4c06640 1 -- 172.21.15.104:0/2473091276 wait complete. 2024-03-29T11:28:16.713 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.706+0000 7fa1f4c06640 10 monclient: build_initial_monmap 2024-03-29T11:28:16.713 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.706+0000 7fa1f4c06640 10 monclient: monmap: 2024-03-29T11:28:16.713 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:28:16.714 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:16.714 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:28:16.714 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:28:16.714 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:16.714 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:28:16.714 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:28:16.714 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:28:16.714 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:28:16.714 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:28:16.714 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.707+0000 7fa1f4c06640 1 librados: starting msgr at 2024-03-29T11:28:16.714 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.707+0000 7fa1f4c06640 1 librados: starting objecter 2024-03-29T11:28:16.715 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.707+0000 7fa1f4c06640 1 Processor -- start 2024-03-29T11:28:16.715 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.707+0000 7fa1f4c06640 1 -- start start 2024-03-29T11:28:16.715 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.707+0000 7fa1f4c06640 1 librados: setting wanted keys 2024-03-29T11:28:16.715 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.707+0000 7fa1f4c06640 1 librados: calling monclient init 2024-03-29T11:28:16.715 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.708+0000 7fa1f4c06640 10 monclient: init 2024-03-29T11:28:16.715 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.708+0000 7fa1f4c06640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:16.715 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.708+0000 7fa1f4c06640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:28:16.715 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.708+0000 7fa1f4c06640 10 monclient(hunting): picked mon.noname-b con 0x7fa1f00a9d70 addr v1:172.21.15.119:6789/0 2024-03-29T11:28:16.715 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.708+0000 7fa1f4c06640 10 monclient(hunting): picked mon.noname-a con 0x7fa1f013f650 addr v1:172.21.15.104:6789/0 2024-03-29T11:28:16.715 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.708+0000 7fa1f4c06640 10 monclient(hunting): picked mon.noname-c con 0x7fa1f01457f0 addr v1:172.21.15.104:6790/0 2024-03-29T11:28:16.715 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.708+0000 7fa1f4c06640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f00ac5b0 con 0x7fa1f013f650 2024-03-29T11:28:16.715 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.708+0000 7fa1f4c06640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f0142b70 con 0x7fa1f00a9d70 2024-03-29T11:28:16.716 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.708+0000 7fa1f4c06640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f0141d70 con 0x7fa1f01457f0 2024-03-29T11:28:16.716 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.708+0000 7fa1f4c06640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:16.716 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.708+0000 7fa1f4c06640 10 monclient(hunting): authenticate will time out at 1477.959612s 2024-03-29T11:28:16.716 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.708+0000 7fa1eed9d640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7fa1f01457f0 0x7fa1f01d7640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:34012/0 (socket says 172.21.15.104:34012) 2024-03-29T11:28:16.716 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.708+0000 7fa1eed9d640 1 -- 172.21.15.104:0/4050312894 learned_addr learned my addr 172.21.15.104:0/4050312894 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:28:16.716 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.708+0000 7fa1deffd640 10 client.?.objecter ms_handle_connect 0x7fa1f01457f0 2024-03-29T11:28:16.716 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.708+0000 7fa1deffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:16.716 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.708+0000 7fa1deffd640 1 -- 172.21.15.104:0/4050312894 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4181571672 0 0) 0x7fa1f0141d70 con 0x7fa1f01457f0 2024-03-29T11:28:16.716 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.708+0000 7fa1deffd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:16.716 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.708+0000 7fa1deffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:16.716 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.708+0000 7fa1deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:16.716 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.708+0000 7fa1deffd640 10 monclient(hunting): my global_id is 4577 2024-03-29T11:28:16.717 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.708+0000 7fa1deffd640 1 -- 172.21.15.104:0/4050312894 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1d0004eb0 con 0x7fa1f01457f0 2024-03-29T11:28:16.717 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.709+0000 7fa1deffd640 1 -- 172.21.15.104:0/4050312894 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1190034268 0 0) 0x7fa1d0004eb0 con 0x7fa1f01457f0 2024-03-29T11:28:16.717 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.709+0000 7fa1deffd640 1 -- 172.21.15.104:0/4050312894 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1f0141d70 con 0x7fa1f01457f0 2024-03-29T11:28:16.717 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.709+0000 7fa1deffd640 1 -- 172.21.15.104:0/4050312894 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fa1e0004000 con 0x7fa1f01457f0 2024-03-29T11:28:16.717 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.709+0000 7fa1deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:16.717 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.709+0000 7fa1deffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:16.717 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.709+0000 7fa1deffd640 10 monclient(hunting): dump: 2024-03-29T11:28:16.717 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:28:16.717 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:28:16.718 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:16.718 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:16.718 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:16.718 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:28:16.718 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:28:16.718 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:28:16.718 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:28:16.718 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:28:16.719 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.709+0000 7fa1deffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:16.719 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.709+0000 7fa1deffd640 1 -- 172.21.15.104:0/4050312894 <== mon.2 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3062602857 0 0) 0x7fa1f0141d70 con 0x7fa1f01457f0 2024-03-29T11:28:16.719 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.709+0000 7fa1deffd640 1 -- 172.21.15.104:0/4050312894 >> v1:172.21.15.119:6789/0 conn(0x7fa1f00a9d70 legacy=0x7fa1f00b0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:16.719 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.709+0000 7fa1deffd640 1 -- 172.21.15.104:0/4050312894 >> v1:172.21.15.104:6789/0 conn(0x7fa1f013f650 legacy=0x7fa1f01d6f30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:16.719 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.710+0000 7fa1deffd640 10 monclient: _finish_hunting 0 2024-03-29T11:28:16.719 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.710+0000 7fa1deffd640 1 monclient: found mon.c 2024-03-29T11:28:16.719 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.710+0000 7fa1deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:16.719 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.710+0000 7fa1deffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:28:16.719 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.710+0000 7fa1deffd640 1 -- 172.21.15.104:0/4050312894 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f0141f00 con 0x7fa1f01457f0 2024-03-29T11:28:16.719 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.710+0000 7fa1deffd640 10 monclient: _finish_auth 0 2024-03-29T11:28:16.719 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.710+0000 7fa1deffd640 10 monclient: _check_auth_tickets 2024-03-29T11:28:16.719 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.710+0000 7fa1deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:16.720 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.710+0000 7fa1f4c06640 5 monclient: authenticate success, global_id 4577 2024-03-29T11:28:16.720 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.710+0000 7fa1f4c06640 10 monclient: _renew_subs 2024-03-29T11:28:16.720 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.710+0000 7fa1f4c06640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:28:16.720 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.710+0000 7fa1f4c06640 1 -- 172.21.15.104:0/4050312894 --> v1:172.21.15.104:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1f00b1530 con 0x7fa1f01457f0 2024-03-29T11:28:16.720 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.710+0000 7fa1f4c06640 10 client.4577.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:16.720 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.710+0000 7fa1f4c06640 10 monclient: _renew_subs 2024-03-29T11:28:16.720 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.710+0000 7fa1f4c06640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:28:16.720 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.710+0000 7fa1f4c06640 1 -- 172.21.15.104:0/4050312894 --> v1:172.21.15.104:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1f0144950 con 0x7fa1f01457f0 2024-03-29T11:28:16.720 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.710+0000 7fa1f4c06640 1 librados: init done 2024-03-29T11:28:16.720 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.711+0000 7fa1deffd640 1 -- 172.21.15.104:0/4050312894 <== mon.2 v1:172.21.15.104:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa1e0004160 con 0x7fa1f01457f0 2024-03-29T11:28:16.720 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.711+0000 7fa1deffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:16.720 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.711+0000 7fa1deffd640 1 -- 172.21.15.104:0/4050312894 <== mon.2 v1:172.21.15.104:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fa1e0001170 con 0x7fa1f01457f0 2024-03-29T11:28:16.721 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.711+0000 7fa1deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:16.721 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.711+0000 7fa1deffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:16.721 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.711+0000 7fa1deffd640 10 monclient: dump: 2024-03-29T11:28:16.721 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:28:16.721 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:28:16.721 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:16.721 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:16.721 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:16.721 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:28:16.721 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:28:16.721 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:28:16.721 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:28:16.722 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:28:16.722 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.712+0000 7fa1f4c06640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:16.722 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.712+0000 7fa1f4c06640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:16.722 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.712+0000 7fa1f4c06640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:28:16.722 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.712+0000 7fa1f4c06640 1 -- 172.21.15.104:0/4050312894 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1f00b1880 con 0x7fa1f01457f0 2024-03-29T11:28:16.722 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.712+0000 7fa1ecd99640 4 set_mon_vals no callback set 2024-03-29T11:28:16.722 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.713+0000 7fa1deffd640 1 -- 172.21.15.104:0/4050312894 <== mon.2 v1:172.21.15.104:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7fa1e0001350 con 0x7fa1f01457f0 2024-03-29T11:28:16.722 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.713+0000 7fa1deffd640 1 -- 172.21.15.104:0/4050312894 <== mon.2 v1:172.21.15.104:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7fa1e0084ab0 con 0x7fa1f01457f0 2024-03-29T11:28:16.722 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.713+0000 7fa1deffd640 10 client.4577.objecter ms_dispatch 0x7fa1f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:16.722 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.713+0000 7fa1deffd640 3 client.4577.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:16.722 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.713+0000 7fa1ee59c640 10 monclient: get_auth_request con 0x7fa1d006dc90 auth_method 0 2024-03-29T11:28:16.722 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.713+0000 7fa1deffd640 3 client.4577.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:16.723 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.713+0000 7fa1deffd640 20 client.4577.objecter dump_active .. 0 homeless 2024-03-29T11:28:16.723 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.715+0000 7fa1deffd640 10 client.4577.objecter ms_handle_connect 0x7fa1d006dc90 2024-03-29T11:28:16.723 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.719+0000 7fa1deffd640 1 -- 172.21.15.104:0/4050312894 <== mon.2 v1:172.21.15.104:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa1e00503f0 con 0x7fa1f01457f0 2024-03-29T11:28:16.723 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.719+0000 7fa1deffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:16.723 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:16.719+0000 7fa1deffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:17.012 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:17.021+0000 7fa1f4c06640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:17.012 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:17.022+0000 7fa1f4c06640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:17.012 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:17.022+0000 7fa1f4c06640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:28:17.012 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:17.022+0000 7fa1f4c06640 1 -- 172.21.15.104:0/4050312894 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1f0075f10 con 0x7fa1f01457f0 2024-03-29T11:28:17.012 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:17.022+0000 7fa1deffd640 1 -- 172.21.15.104:0/4050312894 <== mon.2 v1:172.21.15.104:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7fa1e0052320 con 0x7fa1f01457f0 2024-03-29T11:28:17.012 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:17.022+0000 7fa1deffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:17.012 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:17.022+0000 7fa1deffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:17.013 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:28:17.013 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:17.014 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:17.025+0000 7fa1f4c06640 10 librados: watch_flush enter 2024-03-29T11:28:17.015 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:17.025+0000 7fa1f4c06640 10 librados: watch_flush exit 2024-03-29T11:28:17.015 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:17.025+0000 7fa1f4c06640 20 client.4577.objecter shutdown clearing up homeless session... 2024-03-29T11:28:17.015 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:17.025+0000 7fa1f4c06640 10 client.4577.objecter successfully canceled tick 2024-03-29T11:28:17.015 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:17.025+0000 7fa1f4c06640 1 -- 172.21.15.104:0/4050312894 >> v1:172.21.15.104:6816/3341126660 conn(0x7fa1d006dc90 legacy=0x7fa1d0070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:17.015 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:17.025+0000 7fa1f4c06640 10 monclient: shutdown 2024-03-29T11:28:17.015 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:17.025+0000 7fa1f4c06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:17.015 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:17.025+0000 7fa1f4c06640 1 -- 172.21.15.104:0/4050312894 >> v1:172.21.15.104:6790/0 conn(0x7fa1f01457f0 legacy=0x7fa1f01d7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:17.015 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:17.025+0000 7fa1f4c06640 1 -- 172.21.15.104:0/4050312894 shutdown_connections 2024-03-29T11:28:17.015 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:17.025+0000 7fa1f4c06640 1 -- 172.21.15.104:0/4050312894 >> 172.21.15.104:0/4050312894 conn(0x7fa1f0066a00 msgr2=0x7fa1f00b4920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:17.015 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:17.025+0000 7fa1f4c06640 1 -- 172.21.15.104:0/4050312894 shutdown_connections 2024-03-29T11:28:17.015 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:17.025+0000 7fa1f4c06640 1 -- 172.21.15.104:0/4050312894 wait complete. 2024-03-29T11:28:17.015 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:17.025+0000 7fa1f4c06640 1 librados: shutdown 2024-03-29T11:28:17.028 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:23.836 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:23.985 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:23.994+0000 7f867224e640 10 monclient: get_monmap_and_config 2024-03-29T11:28:23.985 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:23.994+0000 7f867224e640 10 monclient: build_initial_monmap 2024-03-29T11:28:23.985 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:23.994+0000 7f867224e640 10 monclient: monmap: 2024-03-29T11:28:23.986 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:28:23.986 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:28:23.986 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:28:23.996864+0000 2024-03-29T11:28:23.986 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:28:23.996864+0000 2024-03-29T11:28:23.986 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:23.986 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:28:23.986 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:28:23.986 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:28:23.986 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:28:23.986 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:28:23.988 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:23.996+0000 7f867224e640 1 Processor -- start 2024-03-29T11:28:23.988 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:23.996+0000 7f867224e640 1 -- start start 2024-03-29T11:28:23.988 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:23.996+0000 7f867224e640 10 monclient: init 2024-03-29T11:28:23.988 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:23.997+0000 7f867224e640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:23.988 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:23.997+0000 7f867224e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:28:24.012 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:23.997+0000 7f867224e640 10 monclient(hunting): picked mon.noname-a con 0x7f866c1422b0 addr v1:172.21.15.104:6789/0 2024-03-29T11:28:24.012 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:23.997+0000 7f867224e640 10 monclient(hunting): picked mon.noname-c con 0x7f866c142690 addr v1:172.21.15.119:6789/0 2024-03-29T11:28:24.012 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:23.997+0000 7f867224e640 10 monclient(hunting): picked mon.noname-b con 0x7f866c0b17d0 addr v1:172.21.15.104:6790/0 2024-03-29T11:28:24.012 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:23.997+0000 7f867224e640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f866c142c80 con 0x7f866c1422b0 2024-03-29T11:28:24.012 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:23.997+0000 7f867224e640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f866c0aae40 con 0x7f866c142690 2024-03-29T11:28:24.012 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:23.997+0000 7f867224e640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f866c13fbd0 con 0x7f866c0b17d0 2024-03-29T11:28:24.012 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:23.997+0000 7f867224e640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:24.012 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:23.997+0000 7f867224e640 10 monclient(hunting): authenticate will time out at 1485.249267s 2024-03-29T11:28:24.012 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:23.997+0000 7f866affd640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f866c142690 0x7f866c0b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:37788/0 (socket says 172.21.15.104:37788) 2024-03-29T11:28:24.012 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:23.997+0000 7f866affd640 1 -- 172.21.15.104:0/183454664 learned_addr learned my addr 172.21.15.104:0/183454664 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:28:24.013 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:23.998+0000 7f866a7fc640 1 -- 172.21.15.104:0/183454664 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3085394201 0 0) 0x7f866c0aae40 con 0x7f866c142690 2024-03-29T11:28:24.013 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:23.998+0000 7f866a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:28:24.013 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:23.998+0000 7f866a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:24.013 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:23.998+0000 7f866a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:24.013 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:23.998+0000 7f866a7fc640 10 monclient(hunting): my global_id is 4525 2024-03-29T11:28:24.013 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:23.998+0000 7f866a7fc640 1 -- 172.21.15.104:0/183454664 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8654003520 con 0x7f866c142690 2024-03-29T11:28:24.014 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:23.999+0000 7f866a7fc640 1 -- 172.21.15.104:0/183454664 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2574618015 0 0) 0x7f8654003520 con 0x7f866c142690 2024-03-29T11:28:24.014 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.000+0000 7f866a7fc640 1 -- 172.21.15.104:0/183454664 >> v1:172.21.15.104:6790/0 conn(0x7f866c0b17d0 legacy=0x7f866c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:24.014 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.000+0000 7f866a7fc640 1 -- 172.21.15.104:0/183454664 >> v1:172.21.15.104:6789/0 conn(0x7f866c1422b0 legacy=0x7f866c13f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:24.014 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.000+0000 7f866a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:28:24.014 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.000+0000 7f866a7fc640 1 monclient: found mon.noname-c 2024-03-29T11:28:24.015 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.000+0000 7f866a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:24.015 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.000+0000 7f866a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.119:6789/0 2024-03-29T11:28:24.015 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.000+0000 7f866a7fc640 1 -- 172.21.15.104:0/183454664 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f866c151940 con 0x7f866c142690 2024-03-29T11:28:24.015 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.000+0000 7f866a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:28:24.015 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.000+0000 7f866a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:28:24.015 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.000+0000 7f866a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:24.016 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.000+0000 7f867224e640 5 monclient: authenticate success, global_id 4525 2024-03-29T11:28:24.016 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.000+0000 7f867224e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:24.016 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.000+0000 7f866a7fc640 1 -- 172.21.15.104:0/183454664 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f8660001690 con 0x7f866c142690 2024-03-29T11:28:24.016 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.000+0000 7f866a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:24.016 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.000+0000 7f866a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:24.016 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.000+0000 7f866a7fc640 10 monclient: dump: 2024-03-29T11:28:24.017 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:28:24.017 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:28:24.017 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:24.017 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:24.017 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:24.017 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:28:24.018 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:28:24.018 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:28:24.018 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:28:24.018 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:28:24.018 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.000+0000 7f867224e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:24.018 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.002+0000 7f866a7fc640 1 -- 172.21.15.104:0/183454664 <== mon.1 v1:172.21.15.119:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86600017f0 con 0x7f866c142690 2024-03-29T11:28:24.019 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.002+0000 7f866a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:24.019 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.002+0000 7f866a7fc640 1 -- 172.21.15.104:0/183454664 <== mon.1 v1:172.21.15.119:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f8660001c40 con 0x7f866c142690 2024-03-29T11:28:24.019 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.002+0000 7f866a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:24.019 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.002+0000 7f866a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:24.019 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.002+0000 7f866a7fc640 10 monclient: dump: 2024-03-29T11:28:24.019 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:28:24.020 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:28:24.020 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:24.020 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:24.020 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:24.020 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:28:24.021 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:28:24.021 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:28:24.021 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:28:24.021 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:28:24.021 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.002+0000 7f867224e640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:24.021 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.002+0000 7f867224e640 4 set_mon_vals no callback set 2024-03-29T11:28:24.021 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.003+0000 7f867224e640 10 monclient: shutdown 2024-03-29T11:28:24.022 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.003+0000 7f867224e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:24.022 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.003+0000 7f867224e640 1 -- 172.21.15.104:0/183454664 >> v1:172.21.15.119:6789/0 conn(0x7f866c142690 legacy=0x7f866c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:24.022 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.003+0000 7f867224e640 1 -- 172.21.15.104:0/183454664 shutdown_connections 2024-03-29T11:28:24.022 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.003+0000 7f867224e640 1 -- 172.21.15.104:0/183454664 >> 172.21.15.104:0/183454664 conn(0x7f866c066a00 msgr2=0x7f866c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:24.022 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.004+0000 7f867224e640 1 -- 172.21.15.104:0/183454664 shutdown_connections 2024-03-29T11:28:24.023 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.004+0000 7f867224e640 1 -- 172.21.15.104:0/183454664 wait complete. 2024-03-29T11:28:24.023 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.005+0000 7f867224e640 10 monclient: build_initial_monmap 2024-03-29T11:28:24.023 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.005+0000 7f867224e640 10 monclient: monmap: 2024-03-29T11:28:24.023 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:28:24.023 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:24.023 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:28:24.024 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:28:24.024 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:24.024 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:28:24.024 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:28:24.024 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:28:24.024 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:28:24.024 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:28:24.025 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.005+0000 7f867224e640 1 librados: starting msgr at 2024-03-29T11:28:24.025 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.005+0000 7f867224e640 1 librados: starting objecter 2024-03-29T11:28:24.025 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.005+0000 7f867224e640 1 Processor -- start 2024-03-29T11:28:24.025 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.006+0000 7f867224e640 1 -- start start 2024-03-29T11:28:24.025 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.006+0000 7f867224e640 1 librados: setting wanted keys 2024-03-29T11:28:24.025 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.006+0000 7f867224e640 1 librados: calling monclient init 2024-03-29T11:28:24.025 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.006+0000 7f867224e640 10 monclient: init 2024-03-29T11:28:24.026 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.006+0000 7f867224e640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:24.026 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.006+0000 7f867224e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:28:24.026 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.006+0000 7f867224e640 10 monclient(hunting): picked mon.noname-b con 0x7f866c0b17d0 addr v1:172.21.15.119:6789/0 2024-03-29T11:28:24.026 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.006+0000 7f867224e640 10 monclient(hunting): picked mon.noname-c con 0x7f866c1422b0 addr v1:172.21.15.104:6790/0 2024-03-29T11:28:24.026 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.006+0000 7f867224e640 10 monclient(hunting): picked mon.noname-a con 0x7f866c142690 addr v1:172.21.15.104:6789/0 2024-03-29T11:28:24.026 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.006+0000 7f867224e640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f866c0b6e90 con 0x7f866c142690 2024-03-29T11:28:24.027 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.006+0000 7f867224e640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f866c0ab920 con 0x7f866c0b17d0 2024-03-29T11:28:24.027 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.006+0000 7f867224e640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f866c13eec0 con 0x7f866c1422b0 2024-03-29T11:28:24.027 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.006+0000 7f867224e640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:24.027 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.006+0000 7f867224e640 10 monclient(hunting): authenticate will time out at 1485.258767s 2024-03-29T11:28:24.027 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.008+0000 7f866b7fe640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f866c0b17d0 0x7f866c12e1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:38004/0 (socket says 172.21.15.104:38004) 2024-03-29T11:28:24.027 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.008+0000 7f866b7fe640 1 -- 172.21.15.104:0/2075048231 learned_addr learned my addr 172.21.15.104:0/2075048231 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:28:24.027 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.008+0000 7f8653fff640 10 client.?.objecter ms_handle_connect 0x7f866c0b17d0 2024-03-29T11:28:24.028 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.008+0000 7f8653fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:24.028 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.008+0000 7f8653fff640 1 -- 172.21.15.104:0/2075048231 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3828999261 0 0) 0x7f866c0ab920 con 0x7f866c0b17d0 2024-03-29T11:28:24.028 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.008+0000 7f8653fff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:24.028 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.008+0000 7f8653fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:24.028 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.008+0000 7f8653fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:24.028 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.008+0000 7f8653fff640 10 monclient(hunting): my global_id is 4528 2024-03-29T11:28:24.029 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.008+0000 7f8653fff640 1 -- 172.21.15.104:0/2075048231 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8658004eb0 con 0x7f866c0b17d0 2024-03-29T11:28:24.031 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.011+0000 7f8653fff640 1 -- 172.21.15.104:0/2075048231 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4216005691 0 0) 0x7f8658004eb0 con 0x7f866c0b17d0 2024-03-29T11:28:24.031 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.011+0000 7f8653fff640 1 -- 172.21.15.104:0/2075048231 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f866c0ab920 con 0x7f866c0b17d0 2024-03-29T11:28:24.031 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.011+0000 7f8653fff640 1 -- 172.21.15.104:0/2075048231 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f865c001960 con 0x7f866c0b17d0 2024-03-29T11:28:24.031 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.011+0000 7f8653fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:24.031 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.011+0000 7f8653fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:24.031 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.011+0000 7f8653fff640 10 monclient(hunting): dump: 2024-03-29T11:28:24.031 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:28:24.031 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:28:24.031 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:24.031 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:24.031 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:24.031 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:28:24.032 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:28:24.032 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:28:24.032 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:28:24.033 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:28:24.033 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.011+0000 7f8653fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:24.033 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.011+0000 7f8653fff640 10 client.?.objecter ms_handle_connect 0x7f866c1422b0 2024-03-29T11:28:24.033 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.011+0000 7f8653fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:24.033 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.011+0000 7f8653fff640 1 -- 172.21.15.104:0/2075048231 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 954782450 0 0) 0x7f866c13eec0 con 0x7f866c1422b0 2024-03-29T11:28:24.033 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.011+0000 7f8653fff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:24.033 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.011+0000 7f8653fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:24.033 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.012+0000 7f8653fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:24.033 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.012+0000 7f8653fff640 10 monclient(hunting): my global_id is 4580 2024-03-29T11:28:24.033 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.012+0000 7f8653fff640 1 -- 172.21.15.104:0/2075048231 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8658004eb0 con 0x7f866c1422b0 2024-03-29T11:28:24.033 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.012+0000 7f8653fff640 1 -- 172.21.15.104:0/2075048231 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2363863924 0 0) 0x7f866c0ab920 con 0x7f866c0b17d0 2024-03-29T11:28:24.033 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.012+0000 7f8653fff640 1 -- 172.21.15.104:0/2075048231 >> v1:172.21.15.104:6790/0 conn(0x7f866c1422b0 legacy=0x7f866c14daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:24.033 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.012+0000 7f8653fff640 1 -- 172.21.15.104:0/2075048231 >> v1:172.21.15.104:6789/0 conn(0x7f866c142690 legacy=0x7f866c1e2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:24.033 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.012+0000 7f8653fff640 10 monclient: _finish_hunting 0 2024-03-29T11:28:24.033 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.012+0000 7f8653fff640 1 monclient: found mon.b 2024-03-29T11:28:24.034 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.012+0000 7f8653fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:24.034 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.012+0000 7f8653fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:28:24.034 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.012+0000 7f8653fff640 1 -- 172.21.15.104:0/2075048231 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f866c0ab540 con 0x7f866c0b17d0 2024-03-29T11:28:24.034 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.012+0000 7f8653fff640 10 monclient: _finish_auth 0 2024-03-29T11:28:24.034 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.012+0000 7f8653fff640 10 monclient: _check_auth_tickets 2024-03-29T11:28:24.034 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.012+0000 7f8653fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:24.034 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.012+0000 7f867224e640 5 monclient: authenticate success, global_id 4528 2024-03-29T11:28:24.034 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.012+0000 7f867224e640 10 monclient: _renew_subs 2024-03-29T11:28:24.034 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.012+0000 7f867224e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:28:24.035 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.012+0000 7f867224e640 1 -- 172.21.15.104:0/2075048231 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f866c0abe40 con 0x7f866c0b17d0 2024-03-29T11:28:24.035 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.012+0000 7f867224e640 10 client.4528.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:24.035 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.012+0000 7f867224e640 10 monclient: _renew_subs 2024-03-29T11:28:24.035 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.012+0000 7f867224e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:28:24.035 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.012+0000 7f867224e640 1 -- 172.21.15.104:0/2075048231 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f866c0b72d0 con 0x7f866c0b17d0 2024-03-29T11:28:24.035 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.012+0000 7f867224e640 1 librados: init done 2024-03-29T11:28:24.035 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.013+0000 7f8653fff640 1 -- 172.21.15.104:0/2075048231 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f865c001ac0 con 0x7f866c0b17d0 2024-03-29T11:28:24.035 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.013+0000 7f8653fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:24.035 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.013+0000 7f8653fff640 1 -- 172.21.15.104:0/2075048231 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f865c003a40 con 0x7f866c0b17d0 2024-03-29T11:28:24.035 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.013+0000 7f86697fa640 4 set_mon_vals no callback set 2024-03-29T11:28:24.035 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.013+0000 7f8653fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:24.036 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.013+0000 7f8653fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:24.036 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.013+0000 7f8653fff640 10 monclient: dump: 2024-03-29T11:28:24.036 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:28:24.036 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:28:24.036 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:24.036 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:24.036 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:24.036 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:28:24.036 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:28:24.036 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:28:24.036 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:28:24.036 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:28:24.036 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.014+0000 7f867224e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:24.037 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.014+0000 7f867224e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:24.037 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.014+0000 7f867224e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:28:24.037 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.014+0000 7f867224e640 1 -- 172.21.15.104:0/2075048231 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f864c005080 con 0x7f866c0b17d0 2024-03-29T11:28:24.037 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.014+0000 7f8653fff640 1 -- 172.21.15.104:0/2075048231 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f865c003ba0 con 0x7f866c0b17d0 2024-03-29T11:28:24.037 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.015+0000 7f8653fff640 1 -- 172.21.15.104:0/2075048231 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f865c084af0 con 0x7f866c0b17d0 2024-03-29T11:28:24.037 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.015+0000 7f866affd640 10 monclient: get_auth_request con 0x7f865806dd00 auth_method 0 2024-03-29T11:28:24.037 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.016+0000 7f8653fff640 10 client.4528.objecter ms_dispatch 0x7f866c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:24.037 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.016+0000 7f8653fff640 3 client.4528.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:24.037 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.016+0000 7f8653fff640 3 client.4528.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:24.037 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.016+0000 7f8653fff640 20 client.4528.objecter dump_active .. 0 homeless 2024-03-29T11:28:24.037 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.016+0000 7f8653fff640 10 client.4528.objecter ms_handle_connect 0x7f865806dd00 2024-03-29T11:28:24.038 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.023+0000 7f8653fff640 1 -- 172.21.15.104:0/2075048231 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f865c0503d0 con 0x7f866c0b17d0 2024-03-29T11:28:24.038 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.023+0000 7f8653fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:24.038 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.023+0000 7f8653fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:24.329 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.339+0000 7f867224e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:24.330 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.339+0000 7f867224e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:24.330 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.339+0000 7f867224e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:28:24.330 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.339+0000 7f867224e640 1 -- 172.21.15.104:0/2075048231 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f864c005330 con 0x7f866c0b17d0 2024-03-29T11:28:24.330 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.340+0000 7f8653fff640 1 -- 172.21.15.104:0/2075048231 <== mon.1 v1:172.21.15.119:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f865c052300 con 0x7f866c0b17d0 2024-03-29T11:28:24.330 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.340+0000 7f8653fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:24.330 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.340+0000 7f8653fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:24.332 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:28:24.332 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:24.334 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.343+0000 7f867224e640 10 librados: watch_flush enter 2024-03-29T11:28:24.334 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.343+0000 7f867224e640 10 librados: watch_flush exit 2024-03-29T11:28:24.334 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.343+0000 7f867224e640 20 client.4528.objecter shutdown clearing up homeless session... 2024-03-29T11:28:24.334 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.343+0000 7f867224e640 10 client.4528.objecter successfully canceled tick 2024-03-29T11:28:24.334 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.343+0000 7f867224e640 1 -- 172.21.15.104:0/2075048231 >> v1:172.21.15.104:6816/3341126660 conn(0x7f865806dd00 legacy=0x7f86580701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:24.334 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.343+0000 7f867224e640 10 monclient: shutdown 2024-03-29T11:28:24.335 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.343+0000 7f867224e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:24.335 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.343+0000 7f867224e640 1 -- 172.21.15.104:0/2075048231 >> v1:172.21.15.119:6789/0 conn(0x7f866c0b17d0 legacy=0x7f866c12e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:24.335 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.343+0000 7f867224e640 1 -- 172.21.15.104:0/2075048231 shutdown_connections 2024-03-29T11:28:24.335 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.343+0000 7f867224e640 1 -- 172.21.15.104:0/2075048231 >> 172.21.15.104:0/2075048231 conn(0x7f866c066a00 msgr2=0x7f866c0b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:24.335 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.343+0000 7f867224e640 1 -- 172.21.15.104:0/2075048231 shutdown_connections 2024-03-29T11:28:24.335 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.343+0000 7f867224e640 1 -- 172.21.15.104:0/2075048231 wait complete. 2024-03-29T11:28:24.335 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:24.343+0000 7f867224e640 1 librados: shutdown 2024-03-29T11:28:24.351 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:31.157 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:31.306 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.315+0000 7f0d99718640 10 monclient: get_monmap_and_config 2024-03-29T11:28:31.307 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.315+0000 7f0d99718640 10 monclient: build_initial_monmap 2024-03-29T11:28:31.307 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.315+0000 7f0d99718640 10 monclient: monmap: 2024-03-29T11:28:31.307 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:28:31.307 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:28:31.307 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:28:31.317599+0000 2024-03-29T11:28:31.307 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:28:31.317599+0000 2024-03-29T11:28:31.307 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:31.307 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:28:31.307 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:28:31.307 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:28:31.307 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:28:31.308 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:28:31.310 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.317+0000 7f0d99718640 1 Processor -- start 2024-03-29T11:28:31.310 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.317+0000 7f0d99718640 1 -- start start 2024-03-29T11:28:31.310 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.317+0000 7f0d99718640 10 monclient: init 2024-03-29T11:28:31.310 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.318+0000 7f0d99718640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:31.310 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.318+0000 7f0d99718640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:28:31.310 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.318+0000 7f0d99718640 10 monclient(hunting): picked mon.noname-b con 0x7f0d8c0389e0 addr v1:172.21.15.104:6790/0 2024-03-29T11:28:31.310 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.318+0000 7f0d99718640 10 monclient(hunting): picked mon.noname-c con 0x7f0d8c038dc0 addr v1:172.21.15.119:6789/0 2024-03-29T11:28:31.312 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.320+0000 7f0d99718640 10 monclient(hunting): picked mon.noname-a con 0x7f0d8c030660 addr v1:172.21.15.104:6789/0 2024-03-29T11:28:31.312 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.320+0000 7f0d99718640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d8c0c54c0 con 0x7f0d8c030660 2024-03-29T11:28:31.312 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.320+0000 7f0d99718640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d8c036430 con 0x7f0d8c038dc0 2024-03-29T11:28:31.312 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.320+0000 7f0d99718640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d8c039840 con 0x7f0d8c0389e0 2024-03-29T11:28:31.312 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.320+0000 7f0d99718640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:31.313 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.320+0000 7f0d99718640 10 monclient(hunting): authenticate will time out at 1492.572560s 2024-03-29T11:28:31.313 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.320+0000 7f0d937fe640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f0d8c038dc0 0x7f0d8c0cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:60980/0 (socket says 172.21.15.104:60980) 2024-03-29T11:28:31.313 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.320+0000 7f0d937fe640 1 -- 172.21.15.104:0/3677005922 learned_addr learned my addr 172.21.15.104:0/3677005922 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:28:31.313 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.320+0000 7f0d92ffd640 1 -- 172.21.15.104:0/3677005922 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2301413485 0 0) 0x7f0d8c036430 con 0x7f0d8c038dc0 2024-03-29T11:28:31.313 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.320+0000 7f0d92ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:31.313 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.320+0000 7f0d92ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:31.313 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.320+0000 7f0d92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:31.313 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.320+0000 7f0d92ffd640 10 monclient(hunting): my global_id is 4531 2024-03-29T11:28:31.313 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.320+0000 7f0d92ffd640 1 -- 172.21.15.104:0/3677005922 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d84003520 con 0x7f0d8c038dc0 2024-03-29T11:28:31.313 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.321+0000 7f0d92ffd640 1 -- 172.21.15.104:0/3677005922 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 579706557 0 0) 0x7f0d84003520 con 0x7f0d8c038dc0 2024-03-29T11:28:31.313 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.321+0000 7f0d92ffd640 1 -- 172.21.15.104:0/3677005922 >> v1:172.21.15.104:6790/0 conn(0x7f0d8c0389e0 legacy=0x7f0d8c033520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:31.318 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.324+0000 7f0d92ffd640 1 -- 172.21.15.104:0/3677005922 >> v1:172.21.15.104:6789/0 conn(0x7f0d8c030660 legacy=0x7f0d8c0d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:31.319 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.324+0000 7f0d92ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:28:31.319 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.324+0000 7f0d92ffd640 1 monclient: found mon.noname-c 2024-03-29T11:28:31.319 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.324+0000 7f0d92ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:31.319 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.324+0000 7f0d92ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.119:6789/0 2024-03-29T11:28:31.319 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.324+0000 7f0d92ffd640 1 -- 172.21.15.104:0/3677005922 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d8c035690 con 0x7f0d8c038dc0 2024-03-29T11:28:31.319 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.324+0000 7f0d92ffd640 10 monclient: _finish_auth 0 2024-03-29T11:28:31.319 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.324+0000 7f0d92ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:28:31.319 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.324+0000 7f0d92ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:31.319 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.324+0000 7f0d92ffd640 1 -- 172.21.15.104:0/3677005922 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f0d88001690 con 0x7f0d8c038dc0 2024-03-29T11:28:31.319 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.324+0000 7f0d92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:31.319 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.325+0000 7f0d92ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:31.320 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.325+0000 7f0d92ffd640 10 monclient: dump: 2024-03-29T11:28:31.320 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:28:31.320 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:28:31.320 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:31.320 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:31.320 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:31.320 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:28:31.320 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:28:31.320 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:28:31.320 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:28:31.320 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:28:31.320 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.325+0000 7f0d99718640 5 monclient: authenticate success, global_id 4531 2024-03-29T11:28:31.321 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.325+0000 7f0d99718640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:31.321 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.325+0000 7f0d92ffd640 1 -- 172.21.15.104:0/3677005922 <== mon.1 v1:172.21.15.119:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d88001860 con 0x7f0d8c038dc0 2024-03-29T11:28:31.321 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.325+0000 7f0d92ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:31.321 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.325+0000 7f0d99718640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:31.321 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.325+0000 7f0d99718640 4 set_mon_vals no callback set 2024-03-29T11:28:31.321 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.325+0000 7f0d92ffd640 1 -- 172.21.15.104:0/3677005922 <== mon.1 v1:172.21.15.119:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f0d88001cb0 con 0x7f0d8c038dc0 2024-03-29T11:28:31.321 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.325+0000 7f0d92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:31.321 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.325+0000 7f0d92ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:31.322 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.325+0000 7f0d92ffd640 10 monclient: dump: 2024-03-29T11:28:31.322 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:28:31.322 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:28:31.322 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:31.322 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:31.322 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:31.322 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:28:31.322 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:28:31.322 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:28:31.322 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:28:31.322 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:28:31.322 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.326+0000 7f0d99718640 10 monclient: shutdown 2024-03-29T11:28:31.323 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.326+0000 7f0d99718640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:31.323 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.326+0000 7f0d99718640 1 -- 172.21.15.104:0/3677005922 >> v1:172.21.15.119:6789/0 conn(0x7f0d8c038dc0 legacy=0x7f0d8c0cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:31.323 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.326+0000 7f0d99718640 1 -- 172.21.15.104:0/3677005922 shutdown_connections 2024-03-29T11:28:31.323 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.326+0000 7f0d99718640 1 -- 172.21.15.104:0/3677005922 >> 172.21.15.104:0/3677005922 conn(0x7f0d8c005c00 msgr2=0x7f0d8c006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:31.323 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.326+0000 7f0d99718640 1 -- 172.21.15.104:0/3677005922 shutdown_connections 2024-03-29T11:28:31.323 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.326+0000 7f0d99718640 1 -- 172.21.15.104:0/3677005922 wait complete. 2024-03-29T11:28:31.323 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.327+0000 7f0d99718640 10 monclient: build_initial_monmap 2024-03-29T11:28:31.323 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.327+0000 7f0d99718640 10 monclient: monmap: 2024-03-29T11:28:31.323 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:28:31.323 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:31.323 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:28:31.323 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:28:31.324 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:31.324 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:28:31.324 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:28:31.324 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:28:31.324 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:28:31.324 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:28:31.324 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.328+0000 7f0d99718640 1 librados: starting msgr at 2024-03-29T11:28:31.324 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.328+0000 7f0d99718640 1 librados: starting objecter 2024-03-29T11:28:31.324 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.328+0000 7f0d99718640 1 Processor -- start 2024-03-29T11:28:31.325 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.328+0000 7f0d99718640 1 -- start start 2024-03-29T11:28:31.326 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.328+0000 7f0d99718640 1 librados: setting wanted keys 2024-03-29T11:28:31.326 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.328+0000 7f0d99718640 1 librados: calling monclient init 2024-03-29T11:28:31.326 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.328+0000 7f0d99718640 10 monclient: init 2024-03-29T11:28:31.326 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.329+0000 7f0d99718640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:31.326 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.329+0000 7f0d99718640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:28:31.326 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.329+0000 7f0d99718640 10 monclient(hunting): picked mon.noname-c con 0x7f0d8c030660 addr v1:172.21.15.104:6790/0 2024-03-29T11:28:31.326 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.330+0000 7f0d99718640 10 monclient(hunting): picked mon.noname-b con 0x7f0d8c0389e0 addr v1:172.21.15.119:6789/0 2024-03-29T11:28:31.326 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.330+0000 7f0d99718640 10 monclient(hunting): picked mon.noname-a con 0x7f0d8c0c30c0 addr v1:172.21.15.104:6789/0 2024-03-29T11:28:31.326 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.330+0000 7f0d937fe640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f0d8c0389e0 0x7f0d8c164aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:34996/0 (socket says 172.21.15.104:34996) 2024-03-29T11:28:31.326 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.330+0000 7f0d937fe640 1 -- 172.21.15.104:0/3679097914 learned_addr learned my addr 172.21.15.104:0/3679097914 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:28:31.326 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.330+0000 7f0d99718640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d8c009cf0 con 0x7f0d8c0c30c0 2024-03-29T11:28:31.327 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.332+0000 7f0d99718640 1 -- 172.21.15.104:0/3679097914 --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d8c0c3690 con 0x7f0d8c0389e0 2024-03-29T11:28:31.327 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.332+0000 7f0d90ff9640 10 client.?.objecter ms_handle_connect 0x7f0d8c0389e0 2024-03-29T11:28:31.327 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.332+0000 7f0d90ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:31.327 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.332+0000 7f0d99718640 1 -- 172.21.15.104:0/3679097914 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d8c0c5650 con 0x7f0d8c030660 2024-03-29T11:28:31.327 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.332+0000 7f0d99718640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:31.327 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.332+0000 7f0d99718640 10 monclient(hunting): authenticate will time out at 1492.584136s 2024-03-29T11:28:31.327 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.332+0000 7f0d90ff9640 1 -- 172.21.15.104:0/3679097914 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1389175761 0 0) 0x7f0d8c0c3690 con 0x7f0d8c0389e0 2024-03-29T11:28:31.327 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.332+0000 7f0d90ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:28:31.327 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.332+0000 7f0d90ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:31.327 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.332+0000 7f0d90ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:31.327 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.332+0000 7f0d90ff9640 10 monclient(hunting): my global_id is 4534 2024-03-29T11:28:31.327 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.332+0000 7f0d90ff9640 1 -- 172.21.15.104:0/3679097914 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d78004eb0 con 0x7f0d8c0389e0 2024-03-29T11:28:31.328 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.333+0000 7f0d90ff9640 1 -- 172.21.15.104:0/3679097914 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 10666414 0 0) 0x7f0d78004eb0 con 0x7f0d8c0389e0 2024-03-29T11:28:31.328 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.333+0000 7f0d90ff9640 1 -- 172.21.15.104:0/3679097914 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d8c0c3690 con 0x7f0d8c0389e0 2024-03-29T11:28:31.328 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.333+0000 7f0d90ff9640 1 -- 172.21.15.104:0/3679097914 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f0d88001c30 con 0x7f0d8c0389e0 2024-03-29T11:28:31.328 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.333+0000 7f0d90ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:31.328 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.333+0000 7f0d90ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:31.328 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.333+0000 7f0d90ff9640 10 monclient(hunting): dump: 2024-03-29T11:28:31.328 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:28:31.328 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:28:31.328 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:31.328 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:31.328 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:31.328 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:28:31.328 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:28:31.329 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:28:31.329 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:28:31.329 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:28:31.329 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.333+0000 7f0d90ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:31.329 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.333+0000 7f0d90ff9640 1 -- 172.21.15.104:0/3679097914 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4047210193 0 0) 0x7f0d8c0c3690 con 0x7f0d8c0389e0 2024-03-29T11:28:31.329 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.333+0000 7f0d90ff9640 1 -- 172.21.15.104:0/3679097914 >> v1:172.21.15.104:6790/0 conn(0x7f0d8c030660 legacy=0x7f0d8c0c29b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:31.330 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.336+0000 7f0d90ff9640 1 -- 172.21.15.104:0/3679097914 >> v1:172.21.15.104:6789/0 conn(0x7f0d8c0c30c0 legacy=0x7f0d8c1651b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:31.330 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.336+0000 7f0d90ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:28:31.330 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.336+0000 7f0d90ff9640 1 monclient: found mon.b 2024-03-29T11:28:31.330 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.336+0000 7f0d90ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:31.330 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.336+0000 7f0d90ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:28:31.330 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.336+0000 7f0d90ff9640 1 -- 172.21.15.104:0/3679097914 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d8c0c57e0 con 0x7f0d8c0389e0 2024-03-29T11:28:31.330 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.336+0000 7f0d90ff9640 10 monclient: _finish_auth 0 2024-03-29T11:28:31.331 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.336+0000 7f0d90ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:28:31.331 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.336+0000 7f0d90ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:31.331 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.336+0000 7f0d90ff9640 10 client.?.objecter ms_handle_connect 0x7f0d8c0c30c0 2024-03-29T11:28:31.331 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.336+0000 7f0d90ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:31.331 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.336+0000 7f0d99718640 5 monclient: authenticate success, global_id 4534 2024-03-29T11:28:31.331 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.336+0000 7f0d99718640 10 monclient: _renew_subs 2024-03-29T11:28:31.331 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.336+0000 7f0d99718640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:28:31.331 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.336+0000 7f0d99718640 1 -- 172.21.15.104:0/3679097914 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d8c0c8560 con 0x7f0d8c0389e0 2024-03-29T11:28:31.331 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.336+0000 7f0d99718640 10 client.4534.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:31.331 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.336+0000 7f0d99718640 10 monclient: _renew_subs 2024-03-29T11:28:31.331 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.336+0000 7f0d99718640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:28:31.331 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.336+0000 7f0d99718640 1 -- 172.21.15.104:0/3679097914 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d8c0c7c10 con 0x7f0d8c0389e0 2024-03-29T11:28:31.332 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.336+0000 7f0d99718640 1 librados: init done 2024-03-29T11:28:31.332 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.336+0000 7f0d90ff9640 1 -- 172.21.15.104:0/3679097914 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d88004b00 con 0x7f0d8c0389e0 2024-03-29T11:28:31.332 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.336+0000 7f0d90ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:31.332 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.336+0000 7f0d927fc640 4 set_mon_vals no callback set 2024-03-29T11:28:31.332 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.337+0000 7f0d90ff9640 1 -- 172.21.15.104:0/3679097914 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f0d88004f50 con 0x7f0d8c0389e0 2024-03-29T11:28:31.332 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.337+0000 7f0d90ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:31.332 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.337+0000 7f0d90ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:31.332 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.337+0000 7f0d90ff9640 10 monclient: dump: 2024-03-29T11:28:31.332 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:28:31.332 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:28:31.332 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:31.332 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:31.333 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:31.333 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:28:31.333 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:28:31.333 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:28:31.333 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:28:31.333 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:28:31.333 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.337+0000 7f0d7e7fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:31.333 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.337+0000 7f0d7e7fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:31.333 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.337+0000 7f0d7e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:28:31.333 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.337+0000 7f0d7e7fc640 1 -- 172.21.15.104:0/3679097914 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d8c007b90 con 0x7f0d8c0389e0 2024-03-29T11:28:31.334 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.340+0000 7f0d90ff9640 1 -- 172.21.15.104:0/3679097914 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f0d880050b0 con 0x7f0d8c0389e0 2024-03-29T11:28:31.334 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.341+0000 7f0d90ff9640 1 -- 172.21.15.104:0/3679097914 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f0d88084c20 con 0x7f0d8c0389e0 2024-03-29T11:28:31.334 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.341+0000 7f0d90ff9640 10 client.4534.objecter ms_dispatch 0x7f0d94000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:31.334 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.341+0000 7f0d90ff9640 3 client.4534.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:31.335 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.341+0000 7f0d90ff9640 3 client.4534.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:31.335 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.341+0000 7f0d90ff9640 20 client.4534.objecter dump_active .. 0 homeless 2024-03-29T11:28:31.335 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.341+0000 7f0d93fff640 10 monclient: get_auth_request con 0x7f0d7806d9c0 auth_method 0 2024-03-29T11:28:31.335 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.341+0000 7f0d90ff9640 10 client.4534.objecter ms_handle_connect 0x7f0d7806d9c0 2024-03-29T11:28:31.342 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.351+0000 7f0d90ff9640 1 -- 172.21.15.104:0/3679097914 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0d88050500 con 0x7f0d8c0389e0 2024-03-29T11:28:31.342 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.351+0000 7f0d90ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:31.342 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.351+0000 7f0d90ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:31.671 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.680+0000 7f0d7e7fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:31.671 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.680+0000 7f0d7e7fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:31.671 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.680+0000 7f0d7e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:28:31.672 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.680+0000 7f0d7e7fc640 1 -- 172.21.15.104:0/3679097914 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d8c005120 con 0x7f0d8c0389e0 2024-03-29T11:28:31.672 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.682+0000 7f0d90ff9640 1 -- 172.21.15.104:0/3679097914 <== mon.1 v1:172.21.15.119:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f0d88052430 con 0x7f0d8c0389e0 2024-03-29T11:28:31.672 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.682+0000 7f0d90ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:31.672 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.682+0000 7f0d90ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:31.673 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:28:31.673 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:31.675 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.685+0000 7f0d7e7fc640 10 librados: watch_flush enter 2024-03-29T11:28:31.678 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.687+0000 7f0d7e7fc640 10 librados: watch_flush exit 2024-03-29T11:28:31.678 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.687+0000 7f0d7e7fc640 20 client.4534.objecter shutdown clearing up homeless session... 2024-03-29T11:28:31.678 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.687+0000 7f0d7e7fc640 10 client.4534.objecter successfully canceled tick 2024-03-29T11:28:31.679 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.687+0000 7f0d7e7fc640 1 -- 172.21.15.104:0/3679097914 >> v1:172.21.15.104:6816/3341126660 conn(0x7f0d7806d9c0 legacy=0x7f0d7806fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:31.679 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.687+0000 7f0d7e7fc640 10 monclient: shutdown 2024-03-29T11:28:31.679 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.687+0000 7f0d7e7fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:31.679 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.687+0000 7f0d7e7fc640 1 -- 172.21.15.104:0/3679097914 >> v1:172.21.15.119:6789/0 conn(0x7f0d8c0389e0 legacy=0x7f0d8c164aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:31.679 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.689+0000 7f0d7e7fc640 1 -- 172.21.15.104:0/3679097914 shutdown_connections 2024-03-29T11:28:31.679 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.689+0000 7f0d7e7fc640 1 -- 172.21.15.104:0/3679097914 >> 172.21.15.104:0/3679097914 conn(0x7f0d8c005c00 msgr2=0x7f0d8c005fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:31.679 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.689+0000 7f0d7e7fc640 1 -- 172.21.15.104:0/3679097914 shutdown_connections 2024-03-29T11:28:31.680 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.689+0000 7f0d7e7fc640 1 -- 172.21.15.104:0/3679097914 wait complete. 2024-03-29T11:28:31.680 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:31.689+0000 7f0d7e7fc640 1 librados: shutdown 2024-03-29T11:28:31.698 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:38.506 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:38.661 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.671+0000 7f60fd723640 10 monclient: get_monmap_and_config 2024-03-29T11:28:38.662 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.671+0000 7f60fd723640 10 monclient: build_initial_monmap 2024-03-29T11:28:38.662 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.671+0000 7f60fd723640 10 monclient: monmap: 2024-03-29T11:28:38.662 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:28:38.662 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:28:38.662 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:28:38.672530+0000 2024-03-29T11:28:38.662 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:28:38.672530+0000 2024-03-29T11:28:38.662 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:38.662 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:28:38.662 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:28:38.662 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:28:38.662 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:28:38.662 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:28:38.664 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.672+0000 7f60fd723640 1 Processor -- start 2024-03-29T11:28:38.664 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.672+0000 7f60fd723640 1 -- start start 2024-03-29T11:28:38.664 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.672+0000 7f60fd723640 10 monclient: init 2024-03-29T11:28:38.664 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.673+0000 7f60fd723640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:38.664 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.673+0000 7f60fd723640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:28:38.664 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.673+0000 7f60fd723640 10 monclient(hunting): picked mon.noname-a con 0x7f60f81457f0 addr v1:172.21.15.104:6789/0 2024-03-29T11:28:38.664 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.673+0000 7f60fd723640 10 monclient(hunting): picked mon.noname-b con 0x7f60f813f650 addr v1:172.21.15.104:6790/0 2024-03-29T11:28:38.664 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.673+0000 7f60fd723640 10 monclient(hunting): picked mon.noname-c con 0x7f60f80a9d70 addr v1:172.21.15.119:6789/0 2024-03-29T11:28:38.664 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.673+0000 7f60fd723640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60f80aaef0 con 0x7f60f81457f0 2024-03-29T11:28:38.664 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.673+0000 7f60fd723640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60f8145bd0 con 0x7f60f80a9d70 2024-03-29T11:28:38.664 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.673+0000 7f60fd723640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60f8145eb0 con 0x7f60f813f650 2024-03-29T11:28:38.664 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.673+0000 7f60fd723640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:38.665 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.673+0000 7f60fd723640 10 monclient(hunting): authenticate will time out at 1499.924633s 2024-03-29T11:28:38.667 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.678+0000 7f60f67fc640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f60f813f650 0x7f60f80b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:57529/0 (socket says 172.21.15.104:57529) 2024-03-29T11:28:38.667 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.678+0000 7f60f67fc640 1 -- 172.21.15.104:0/3989326899 learned_addr learned my addr 172.21.15.104:0/3989326899 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:28:38.668 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.679+0000 7f60f5ffb640 1 -- 172.21.15.104:0/3989326899 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4179959446 0 0) 0x7f60f8145eb0 con 0x7f60f813f650 2024-03-29T11:28:38.669 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.679+0000 7f60f5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:38.669 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.679+0000 7f60f5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:38.669 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.679+0000 7f60f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:38.669 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.679+0000 7f60f5ffb640 10 monclient(hunting): my global_id is 4586 2024-03-29T11:28:38.669 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.679+0000 7f60f5ffb640 1 -- 172.21.15.104:0/3989326899 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60e0003520 con 0x7f60f813f650 2024-03-29T11:28:38.671 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.681+0000 7f60f5ffb640 1 -- 172.21.15.104:0/3989326899 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 549523209 0 0) 0x7f60e0003520 con 0x7f60f813f650 2024-03-29T11:28:38.671 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.681+0000 7f60f5ffb640 1 -- 172.21.15.104:0/3989326899 >> v1:172.21.15.119:6789/0 conn(0x7f60f80a9d70 legacy=0x7f60f814d8c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:28:38.672 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.682+0000 7f60f5ffb640 1 -- 172.21.15.104:0/3989326899 >> v1:172.21.15.104:6789/0 conn(0x7f60f81457f0 legacy=0x7f60f813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:38.672 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.682+0000 7f60f5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:28:38.672 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.682+0000 7f60f5ffb640 1 monclient: found mon.noname-b 2024-03-29T11:28:38.673 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.682+0000 7f60f5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:38.673 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.682+0000 7f60f5ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.104:6790/0 2024-03-29T11:28:38.673 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.682+0000 7f60f5ffb640 1 -- 172.21.15.104:0/3989326899 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60f8146340 con 0x7f60f813f650 2024-03-29T11:28:38.673 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.682+0000 7f60f5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:28:38.673 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.682+0000 7f60f5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:28:38.673 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.682+0000 7f60f5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:38.673 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.682+0000 7f60fd723640 5 monclient: authenticate success, global_id 4586 2024-03-29T11:28:38.673 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.682+0000 7f60fd723640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:38.674 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.683+0000 7f60f5ffb640 1 -- 172.21.15.104:0/3989326899 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f60ec001690 con 0x7f60f813f650 2024-03-29T11:28:38.674 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.683+0000 7f60f5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:38.674 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.683+0000 7f60f5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:38.675 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.683+0000 7f60f5ffb640 10 monclient: dump: 2024-03-29T11:28:38.675 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:28:38.675 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:28:38.675 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:38.675 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:38.675 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:38.675 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:28:38.676 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:28:38.676 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:28:38.676 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:28:38.676 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:28:38.676 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.683+0000 7f60f5ffb640 1 -- 172.21.15.104:0/3989326899 <== mon.2 v1:172.21.15.104:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f60ec001920 con 0x7f60f813f650 2024-03-29T11:28:38.676 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.683+0000 7f60f5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:38.676 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.683+0000 7f60f5ffb640 1 -- 172.21.15.104:0/3989326899 <== mon.2 v1:172.21.15.104:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f60ec001cf0 con 0x7f60f813f650 2024-03-29T11:28:38.676 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.683+0000 7f60f5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:38.676 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.683+0000 7f60f5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:38.676 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.683+0000 7f60f5ffb640 10 monclient: dump: 2024-03-29T11:28:38.676 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:28:38.677 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:28:38.677 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:38.677 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:38.677 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:38.677 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:28:38.677 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:28:38.677 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:28:38.677 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:28:38.677 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:28:38.677 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.683+0000 7f60fd723640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:38.677 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.683+0000 7f60fd723640 4 set_mon_vals no callback set 2024-03-29T11:28:38.677 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.684+0000 7f60fd723640 10 monclient: shutdown 2024-03-29T11:28:38.677 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.684+0000 7f60fd723640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:38.677 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.684+0000 7f60fd723640 1 -- 172.21.15.104:0/3989326899 >> v1:172.21.15.104:6790/0 conn(0x7f60f813f650 legacy=0x7f60f80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:38.678 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.686+0000 7f60fd723640 1 -- 172.21.15.104:0/3989326899 shutdown_connections 2024-03-29T11:28:38.678 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.686+0000 7f60fd723640 1 -- 172.21.15.104:0/3989326899 >> 172.21.15.104:0/3989326899 conn(0x7f60f8066a00 msgr2=0x7f60f8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:38.678 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.687+0000 7f60fd723640 1 -- 172.21.15.104:0/3989326899 shutdown_connections 2024-03-29T11:28:38.679 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.687+0000 7f60fd723640 1 -- 172.21.15.104:0/3989326899 wait complete. 2024-03-29T11:28:38.679 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.688+0000 7f60fd723640 10 monclient: build_initial_monmap 2024-03-29T11:28:38.679 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.688+0000 7f60fd723640 10 monclient: monmap: 2024-03-29T11:28:38.679 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:28:38.679 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:38.679 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:28:38.679 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:28:38.679 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:38.679 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:28:38.679 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:28:38.679 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:28:38.679 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:28:38.679 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:28:38.679 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.688+0000 7f60fd723640 1 librados: starting msgr at 2024-03-29T11:28:38.680 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.688+0000 7f60fd723640 1 librados: starting objecter 2024-03-29T11:28:38.680 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.689+0000 7f60fd723640 1 Processor -- start 2024-03-29T11:28:38.681 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.689+0000 7f60fd723640 1 -- start start 2024-03-29T11:28:38.681 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.689+0000 7f60fd723640 1 librados: setting wanted keys 2024-03-29T11:28:38.681 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.689+0000 7f60fd723640 1 librados: calling monclient init 2024-03-29T11:28:38.681 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.689+0000 7f60fd723640 10 monclient: init 2024-03-29T11:28:38.682 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.691+0000 7f60fd723640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:38.682 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.691+0000 7f60fd723640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:28:38.682 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.691+0000 7f60fd723640 10 monclient(hunting): picked mon.noname-a con 0x7f60f80a9d70 addr v1:172.21.15.104:6789/0 2024-03-29T11:28:38.682 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.692+0000 7f60fd723640 10 monclient(hunting): picked mon.noname-b con 0x7f60f813f650 addr v1:172.21.15.119:6789/0 2024-03-29T11:28:38.682 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.692+0000 7f60fd723640 10 monclient(hunting): picked mon.noname-c con 0x7f60f81457f0 addr v1:172.21.15.104:6790/0 2024-03-29T11:28:38.682 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.692+0000 7f60fd723640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60f8071170 con 0x7f60f80a9d70 2024-03-29T11:28:38.682 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.692+0000 7f60fd723640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60f806fe40 con 0x7f60f813f650 2024-03-29T11:28:38.682 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.692+0000 7f60fd723640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60f8072010 con 0x7f60f81457f0 2024-03-29T11:28:38.683 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.692+0000 7f60fd723640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:38.683 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.692+0000 7f60fd723640 10 monclient(hunting): authenticate will time out at 1499.943051s 2024-03-29T11:28:38.691 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.696+0000 7f60f67fc640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f60f813f650 0x7f60f8141450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:59571/0 (socket says 172.21.15.104:59571) 2024-03-29T11:28:38.691 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.696+0000 7f60f67fc640 1 -- 172.21.15.104:0/3802168381 learned_addr learned my addr 172.21.15.104:0/3802168381 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:28:38.692 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.696+0000 7f60df7fe640 10 client.?.objecter ms_handle_connect 0x7f60f813f650 2024-03-29T11:28:38.692 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.696+0000 7f60df7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:38.692 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.696+0000 7f60df7fe640 1 -- 172.21.15.104:0/3802168381 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3804707450 0 0) 0x7f60f806fe40 con 0x7f60f813f650 2024-03-29T11:28:38.692 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.697+0000 7f60df7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:28:38.692 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.697+0000 7f60df7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:38.692 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.697+0000 7f60df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:38.692 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.697+0000 7f60df7fe640 10 monclient(hunting): my global_id is 4537 2024-03-29T11:28:38.692 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.697+0000 7f60df7fe640 1 -- 172.21.15.104:0/3802168381 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60e4004eb0 con 0x7f60f813f650 2024-03-29T11:28:38.693 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.697+0000 7f60df7fe640 10 client.?.objecter ms_handle_connect 0x7f60f81457f0 2024-03-29T11:28:38.693 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.697+0000 7f60df7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:38.693 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.697+0000 7f60df7fe640 1 -- 172.21.15.104:0/3802168381 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1539167630 0 0) 0x7f60e4004eb0 con 0x7f60f813f650 2024-03-29T11:28:38.693 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.697+0000 7f60df7fe640 1 -- 172.21.15.104:0/3802168381 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f60f806fe40 con 0x7f60f813f650 2024-03-29T11:28:38.693 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.697+0000 7f60df7fe640 1 -- 172.21.15.104:0/3802168381 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f60ec001610 con 0x7f60f813f650 2024-03-29T11:28:38.693 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.697+0000 7f60df7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:38.693 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.697+0000 7f60df7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:38.693 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.697+0000 7f60df7fe640 10 monclient(hunting): dump: 2024-03-29T11:28:38.693 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:28:38.693 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:28:38.693 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:38.693 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:38.693 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:38.693 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:28:38.693 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:28:38.694 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:28:38.694 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:28:38.694 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:28:38.694 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.697+0000 7f60df7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:38.694 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.697+0000 7f60df7fe640 1 -- 172.21.15.104:0/3802168381 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1387306527 0 0) 0x7f60f806fe40 con 0x7f60f813f650 2024-03-29T11:28:38.694 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.697+0000 7f60df7fe640 1 -- 172.21.15.104:0/3802168381 >> v1:172.21.15.104:6790/0 conn(0x7f60f81457f0 legacy=0x7f60f81e6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:38.694 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.697+0000 7f60df7fe640 1 -- 172.21.15.104:0/3802168381 >> v1:172.21.15.104:6789/0 conn(0x7f60f80a9d70 legacy=0x7f60f8071900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:38.694 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.698+0000 7f60df7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:28:38.695 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.698+0000 7f60df7fe640 1 monclient: found mon.b 2024-03-29T11:28:38.695 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.698+0000 7f60df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:38.695 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.698+0000 7f60df7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:28:38.695 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.698+0000 7f60df7fe640 1 -- 172.21.15.104:0/3802168381 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60f8130670 con 0x7f60f813f650 2024-03-29T11:28:38.695 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.698+0000 7f60df7fe640 10 monclient: _finish_auth 0 2024-03-29T11:28:38.695 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.698+0000 7f60df7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:28:38.695 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.698+0000 7f60df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:38.695 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.698+0000 7f60fd723640 5 monclient: authenticate success, global_id 4537 2024-03-29T11:28:38.695 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.698+0000 7f60fd723640 10 monclient: _renew_subs 2024-03-29T11:28:38.695 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.698+0000 7f60fd723640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:28:38.695 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.699+0000 7f60fd723640 1 -- 172.21.15.104:0/3802168381 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60f806f620 con 0x7f60f813f650 2024-03-29T11:28:38.695 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.699+0000 7f60fd723640 10 client.4537.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:38.695 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.699+0000 7f60fd723640 10 monclient: _renew_subs 2024-03-29T11:28:38.696 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.699+0000 7f60fd723640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:28:38.696 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.699+0000 7f60fd723640 1 -- 172.21.15.104:0/3802168381 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f60f806faa0 con 0x7f60f813f650 2024-03-29T11:28:38.696 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.699+0000 7f60fd723640 1 librados: init done 2024-03-29T11:28:38.696 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.699+0000 7f60df7fe640 1 -- 172.21.15.104:0/3802168381 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f60ec004ca0 con 0x7f60f813f650 2024-03-29T11:28:38.696 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.699+0000 7f60df7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:38.696 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.699+0000 7f60f57fa640 4 set_mon_vals no callback set 2024-03-29T11:28:38.696 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.701+0000 7f60df7fe640 1 -- 172.21.15.104:0/3802168381 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f60ec0050b0 con 0x7f60f813f650 2024-03-29T11:28:38.696 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.701+0000 7f60df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:38.696 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.701+0000 7f60df7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:38.696 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.701+0000 7f60df7fe640 10 monclient: dump: 2024-03-29T11:28:38.696 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:28:38.696 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:28:38.697 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:38.697 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:38.697 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:38.697 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:28:38.697 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:28:38.698 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:28:38.698 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:28:38.698 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:28:38.698 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.701+0000 7f60fd723640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:38.698 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.701+0000 7f60fd723640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:38.698 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.701+0000 7f60fd723640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:28:38.698 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.701+0000 7f60fd723640 1 -- 172.21.15.104:0/3802168381 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60d8005080 con 0x7f60f813f650 2024-03-29T11:28:38.699 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.702+0000 7f60df7fe640 1 -- 172.21.15.104:0/3802168381 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f60ec005290 con 0x7f60f813f650 2024-03-29T11:28:38.700 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.703+0000 7f60f6ffd640 10 monclient: get_auth_request con 0x7f60e406dc80 auth_method 0 2024-03-29T11:28:38.700 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.703+0000 7f60df7fe640 1 -- 172.21.15.104:0/3802168381 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f60ec084d40 con 0x7f60f813f650 2024-03-29T11:28:38.700 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.703+0000 7f60df7fe640 10 client.4537.objecter ms_dispatch 0x7f60f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:38.700 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.703+0000 7f60df7fe640 3 client.4537.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:38.700 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.703+0000 7f60df7fe640 3 client.4537.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:38.700 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.703+0000 7f60df7fe640 20 client.4537.objecter dump_active .. 0 homeless 2024-03-29T11:28:38.700 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.703+0000 7f60df7fe640 10 client.4537.objecter ms_handle_connect 0x7f60e406dc80 2024-03-29T11:28:38.701 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.711+0000 7f60df7fe640 1 -- 172.21.15.104:0/3802168381 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f60ec050620 con 0x7f60f813f650 2024-03-29T11:28:38.701 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.711+0000 7f60df7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:38.701 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:38.711+0000 7f60df7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:39.025 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:39.035+0000 7f60fd723640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:39.025 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:39.035+0000 7f60fd723640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:39.025 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:39.035+0000 7f60fd723640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:28:39.026 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:39.035+0000 7f60fd723640 1 -- 172.21.15.104:0/3802168381 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f60d8005330 con 0x7f60f813f650 2024-03-29T11:28:39.032 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:39.040+0000 7f60df7fe640 1 -- 172.21.15.104:0/3802168381 <== mon.1 v1:172.21.15.119:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f60ec052550 con 0x7f60f813f650 2024-03-29T11:28:39.032 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:39.040+0000 7f60df7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:39.032 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:39.040+0000 7f60df7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:39.033 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:28:39.033 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:39.036 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:39.046+0000 7f60fd723640 10 librados: watch_flush enter 2024-03-29T11:28:39.036 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:39.046+0000 7f60fd723640 10 librados: watch_flush exit 2024-03-29T11:28:39.036 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:39.046+0000 7f60fd723640 20 client.4537.objecter shutdown clearing up homeless session... 2024-03-29T11:28:39.036 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:39.046+0000 7f60fd723640 10 client.4537.objecter successfully canceled tick 2024-03-29T11:28:39.036 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:39.046+0000 7f60fd723640 1 -- 172.21.15.104:0/3802168381 >> v1:172.21.15.104:6816/3341126660 conn(0x7f60e406dc80 legacy=0x7f60e4070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:39.036 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:39.046+0000 7f60fd723640 10 monclient: shutdown 2024-03-29T11:28:39.036 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:39.046+0000 7f60fd723640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:39.036 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:39.046+0000 7f60fd723640 1 -- 172.21.15.104:0/3802168381 >> v1:172.21.15.119:6789/0 conn(0x7f60f813f650 legacy=0x7f60f8141450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:39.039 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:39.049+0000 7f60fd723640 1 -- 172.21.15.104:0/3802168381 shutdown_connections 2024-03-29T11:28:39.039 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:39.049+0000 7f60fd723640 1 -- 172.21.15.104:0/3802168381 >> 172.21.15.104:0/3802168381 conn(0x7f60f8066a00 msgr2=0x7f60f80b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:39.039 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:39.049+0000 7f60fd723640 1 -- 172.21.15.104:0/3802168381 shutdown_connections 2024-03-29T11:28:39.040 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:39.049+0000 7f60fd723640 1 -- 172.21.15.104:0/3802168381 wait complete. 2024-03-29T11:28:39.040 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:39.049+0000 7f60fd723640 1 librados: shutdown 2024-03-29T11:28:39.057 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:45.865 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:46.012 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.021+0000 7f379bfff640 10 monclient: get_monmap_and_config 2024-03-29T11:28:46.012 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.021+0000 7f379bfff640 10 monclient: build_initial_monmap 2024-03-29T11:28:46.012 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.021+0000 7f379bfff640 10 monclient: monmap: 2024-03-29T11:28:46.012 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:28:46.012 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:28:46.012 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:28:46.021988+0000 2024-03-29T11:28:46.012 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:28:46.021988+0000 2024-03-29T11:28:46.012 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:46.013 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:28:46.013 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:28:46.013 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:28:46.013 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:28:46.013 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:28:46.014 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.022+0000 7f379bfff640 1 Processor -- start 2024-03-29T11:28:46.014 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.022+0000 7f379bfff640 1 -- start start 2024-03-29T11:28:46.015 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.022+0000 7f379bfff640 10 monclient: init 2024-03-29T11:28:46.015 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.023+0000 7f379bfff640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:46.015 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.023+0000 7f379bfff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:28:46.015 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.023+0000 7f379bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f379c0a9750 addr v1:172.21.15.104:6789/0 2024-03-29T11:28:46.015 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.023+0000 7f379bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f379c0adca0 addr v1:172.21.15.119:6789/0 2024-03-29T11:28:46.015 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.023+0000 7f379bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f379c146a80 addr v1:172.21.15.104:6790/0 2024-03-29T11:28:46.015 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.023+0000 7f379bfff640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f379c0abdc0 con 0x7f379c0a9750 2024-03-29T11:28:46.015 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.023+0000 7f379bfff640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f379c0b0070 con 0x7f379c0adca0 2024-03-29T11:28:46.015 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.023+0000 7f379bfff640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f379c0af880 con 0x7f379c146a80 2024-03-29T11:28:46.015 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.023+0000 7f379bfff640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:46.015 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.023+0000 7f379bfff640 10 monclient(hunting): authenticate will time out at 1507.274217s 2024-03-29T11:28:46.015 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.024+0000 7f379b7fe640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f379c146a80 0x7f379c151bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:57402/0 (socket says 172.21.15.104:57402) 2024-03-29T11:28:46.015 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.024+0000 7f379b7fe640 1 -- 172.21.15.104:0/3261335604 learned_addr learned my addr 172.21.15.104:0/3261335604 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:28:46.018 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.026+0000 7f3799ffb640 1 -- 172.21.15.104:0/3261335604 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 65541694 0 0) 0x7f379c0af880 con 0x7f379c146a80 2024-03-29T11:28:46.018 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.026+0000 7f3799ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:46.018 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.026+0000 7f3799ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:46.018 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.026+0000 7f3799ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:46.018 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.026+0000 7f3799ffb640 10 monclient(hunting): my global_id is 4589 2024-03-29T11:28:46.018 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.026+0000 7f3799ffb640 1 -- 172.21.15.104:0/3261335604 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3784003520 con 0x7f379c146a80 2024-03-29T11:28:46.018 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.027+0000 7f3799ffb640 1 -- 172.21.15.104:0/3261335604 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4247545532 0 0) 0x7f379c0b0070 con 0x7f379c0adca0 2024-03-29T11:28:46.018 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.027+0000 7f3799ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:46.018 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.027+0000 7f3799ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:46.019 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.027+0000 7f3799ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:46.019 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.027+0000 7f3799ffb640 10 monclient(hunting): my global_id is 4540 2024-03-29T11:28:46.019 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.027+0000 7f3799ffb640 1 -- 172.21.15.104:0/3261335604 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f379c0af880 con 0x7f379c0adca0 2024-03-29T11:28:46.019 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.027+0000 7f3799ffb640 1 -- 172.21.15.104:0/3261335604 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2429870139 0 0) 0x7f3784003520 con 0x7f379c146a80 2024-03-29T11:28:46.019 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.027+0000 7f3799ffb640 1 -- 172.21.15.104:0/3261335604 >> v1:172.21.15.119:6789/0 conn(0x7f379c0adca0 legacy=0x7f379c146370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:46.019 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.027+0000 7f3799ffb640 1 -- 172.21.15.104:0/3261335604 >> v1:172.21.15.104:6789/0 conn(0x7f379c0a9750 legacy=0x7f379c143710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:46.019 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.027+0000 7f3799ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:28:46.020 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.027+0000 7f3799ffb640 1 monclient: found mon.noname-b 2024-03-29T11:28:46.020 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.027+0000 7f3799ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:46.020 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.027+0000 7f3799ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.104:6790/0 2024-03-29T11:28:46.020 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.027+0000 7f3799ffb640 1 -- 172.21.15.104:0/3261335604 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f379c0ac030 con 0x7f379c146a80 2024-03-29T11:28:46.020 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.027+0000 7f3799ffb640 10 monclient: _finish_auth 0 2024-03-29T11:28:46.020 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.027+0000 7f3799ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:28:46.020 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.027+0000 7f3799ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:46.020 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.027+0000 7f3799ffb640 1 -- 172.21.15.104:0/3261335604 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f378c001a10 con 0x7f379c146a80 2024-03-29T11:28:46.020 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.027+0000 7f3799ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:46.020 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.027+0000 7f3799ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:46.020 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.027+0000 7f3799ffb640 10 monclient: dump: 2024-03-29T11:28:46.020 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:28:46.020 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:28:46.020 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:46.021 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:46.021 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:46.021 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:28:46.021 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:28:46.021 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:28:46.021 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:28:46.021 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:28:46.021 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.027+0000 7f3799ffb640 1 -- 172.21.15.104:0/3261335604 <== mon.2 v1:172.21.15.104:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f378c001bf0 con 0x7f379c146a80 2024-03-29T11:28:46.021 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.027+0000 7f3799ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:46.021 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.027+0000 7f3799ffb640 1 -- 172.21.15.104:0/3261335604 <== mon.2 v1:172.21.15.104:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f378c002000 con 0x7f379c146a80 2024-03-29T11:28:46.021 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.027+0000 7f3799ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:46.021 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.027+0000 7f3799ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:46.021 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.027+0000 7f3799ffb640 10 monclient: dump: 2024-03-29T11:28:46.021 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:28:46.021 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:28:46.022 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:46.022 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:46.022 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:46.022 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:28:46.022 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:28:46.022 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:28:46.022 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:28:46.022 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:28:46.028 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.028+0000 7f379bfff640 5 monclient: authenticate success, global_id 4589 2024-03-29T11:28:46.028 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.028+0000 7f379bfff640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:46.029 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.028+0000 7f379bfff640 4 set_mon_vals no callback set 2024-03-29T11:28:46.029 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.029+0000 7f379bfff640 10 monclient: shutdown 2024-03-29T11:28:46.029 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.029+0000 7f379bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:46.029 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.029+0000 7f379bfff640 1 -- 172.21.15.104:0/3261335604 >> v1:172.21.15.104:6790/0 conn(0x7f379c146a80 legacy=0x7f379c151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:46.029 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.029+0000 7f379bfff640 1 -- 172.21.15.104:0/3261335604 shutdown_connections 2024-03-29T11:28:46.029 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.029+0000 7f379bfff640 1 -- 172.21.15.104:0/3261335604 >> 172.21.15.104:0/3261335604 conn(0x7f379c0673e0 msgr2=0x7f379c0b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:46.029 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.029+0000 7f379bfff640 1 -- 172.21.15.104:0/3261335604 shutdown_connections 2024-03-29T11:28:46.029 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.029+0000 7f379bfff640 1 -- 172.21.15.104:0/3261335604 wait complete. 2024-03-29T11:28:46.029 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.030+0000 7f379bfff640 10 monclient: build_initial_monmap 2024-03-29T11:28:46.029 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.030+0000 7f379bfff640 10 monclient: monmap: 2024-03-29T11:28:46.029 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:28:46.029 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:46.029 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:28:46.029 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:28:46.030 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:46.030 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:28:46.030 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:28:46.030 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:28:46.030 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:28:46.030 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:28:46.030 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.030+0000 7f379bfff640 1 librados: starting msgr at 2024-03-29T11:28:46.030 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.030+0000 7f379bfff640 1 librados: starting objecter 2024-03-29T11:28:46.030 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.030+0000 7f379bfff640 1 Processor -- start 2024-03-29T11:28:46.030 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.031+0000 7f379bfff640 1 -- start start 2024-03-29T11:28:46.030 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.031+0000 7f379bfff640 1 librados: setting wanted keys 2024-03-29T11:28:46.030 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.031+0000 7f379bfff640 1 librados: calling monclient init 2024-03-29T11:28:46.030 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.031+0000 7f379bfff640 10 monclient: init 2024-03-29T11:28:46.030 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.032+0000 7f379bfff640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:46.030 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.032+0000 7f379bfff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:28:46.031 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.032+0000 7f379bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f379c0adca0 addr v1:172.21.15.119:6789/0 2024-03-29T11:28:46.031 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.032+0000 7f379bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f379c146a80 addr v1:172.21.15.104:6789/0 2024-03-29T11:28:46.031 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.032+0000 7f379bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f379c06e540 addr v1:172.21.15.104:6790/0 2024-03-29T11:28:46.031 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.032+0000 7f379bfff640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f379c0718a0 con 0x7f379c146a80 2024-03-29T11:28:46.031 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.032+0000 7f379bfff640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f379c1de850 con 0x7f379c0adca0 2024-03-29T11:28:46.031 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.032+0000 7f379bfff640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f379c1df6d0 con 0x7f379c06e540 2024-03-29T11:28:46.031 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.032+0000 7f379bfff640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:46.031 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.032+0000 7f379bfff640 10 monclient(hunting): authenticate will time out at 1507.283131s 2024-03-29T11:28:46.031 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.032+0000 7f379b7fe640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f379c06e540 0x7f379c1e5e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:35932/0 (socket says 172.21.15.104:35932) 2024-03-29T11:28:46.031 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.032+0000 7f379b7fe640 1 -- 172.21.15.104:0/3248012035 learned_addr learned my addr 172.21.15.104:0/3248012035 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:28:46.031 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.032+0000 7f3783fff640 10 client.?.objecter ms_handle_connect 0x7f379c06e540 2024-03-29T11:28:46.031 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.032+0000 7f3783fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:46.031 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.033+0000 7f3783fff640 1 -- 172.21.15.104:0/3248012035 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3572178621 0 0) 0x7f379c1df6d0 con 0x7f379c06e540 2024-03-29T11:28:46.031 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.033+0000 7f3783fff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:46.032 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.033+0000 7f3783fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:46.032 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.033+0000 7f3783fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:46.032 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.033+0000 7f3783fff640 10 monclient(hunting): my global_id is 4592 2024-03-29T11:28:46.032 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.033+0000 7f3783fff640 1 -- 172.21.15.104:0/3248012035 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3788004eb0 con 0x7f379c06e540 2024-03-29T11:28:46.032 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.035+0000 7f3783fff640 1 -- 172.21.15.104:0/3248012035 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3393933894 0 0) 0x7f3788004eb0 con 0x7f379c06e540 2024-03-29T11:28:46.032 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.035+0000 7f3783fff640 1 -- 172.21.15.104:0/3248012035 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f379c1df6d0 con 0x7f379c06e540 2024-03-29T11:28:46.032 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.035+0000 7f3783fff640 1 -- 172.21.15.104:0/3248012035 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f378c002000 con 0x7f379c06e540 2024-03-29T11:28:46.032 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.035+0000 7f3783fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:46.032 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.035+0000 7f3783fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:46.032 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.035+0000 7f3783fff640 10 monclient(hunting): dump: 2024-03-29T11:28:46.032 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:28:46.032 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:28:46.032 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:46.032 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:46.032 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:46.033 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:28:46.033 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:28:46.033 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:28:46.033 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:28:46.033 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:28:46.033 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.036+0000 7f3783fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:46.033 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.036+0000 7f3783fff640 10 client.?.objecter ms_handle_connect 0x7f379c0adca0 2024-03-29T11:28:46.033 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.036+0000 7f3783fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:46.033 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.036+0000 7f3783fff640 1 -- 172.21.15.104:0/3248012035 <== mon.2 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 364359795 0 0) 0x7f379c1df6d0 con 0x7f379c06e540 2024-03-29T11:28:46.034 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.036+0000 7f3783fff640 1 -- 172.21.15.104:0/3248012035 >> v1:172.21.15.119:6789/0 conn(0x7f379c0adca0 legacy=0x7f379c071190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:46.034 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.036+0000 7f3783fff640 1 -- 172.21.15.104:0/3248012035 >> v1:172.21.15.104:6789/0 conn(0x7f379c146a80 legacy=0x7f379c1defc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:46.038 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.039+0000 7f3783fff640 10 monclient: _finish_hunting 0 2024-03-29T11:28:46.038 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.039+0000 7f3783fff640 1 monclient: found mon.c 2024-03-29T11:28:46.038 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.039+0000 7f3783fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:46.038 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.039+0000 7f3783fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:28:46.038 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.039+0000 7f3783fff640 1 -- 172.21.15.104:0/3248012035 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f379c071f30 con 0x7f379c06e540 2024-03-29T11:28:46.038 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.039+0000 7f3783fff640 10 monclient: _finish_auth 0 2024-03-29T11:28:46.038 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.039+0000 7f3783fff640 10 monclient: _check_auth_tickets 2024-03-29T11:28:46.038 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.039+0000 7f3783fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:46.038 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.039+0000 7f379bfff640 5 monclient: authenticate success, global_id 4592 2024-03-29T11:28:46.039 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.039+0000 7f379bfff640 10 monclient: _renew_subs 2024-03-29T11:28:46.039 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.039+0000 7f379bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:28:46.039 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.039+0000 7f379bfff640 1 -- 172.21.15.104:0/3248012035 --> v1:172.21.15.104:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f379c072180 con 0x7f379c06e540 2024-03-29T11:28:46.039 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.040+0000 7f379bfff640 10 client.4592.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:46.039 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.040+0000 7f379bfff640 10 monclient: _renew_subs 2024-03-29T11:28:46.039 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.040+0000 7f379bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:28:46.039 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.040+0000 7f379bfff640 1 -- 172.21.15.104:0/3248012035 --> v1:172.21.15.104:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f379c072480 con 0x7f379c06e540 2024-03-29T11:28:46.039 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.040+0000 7f379bfff640 1 librados: init done 2024-03-29T11:28:46.039 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.041+0000 7f3783fff640 1 -- 172.21.15.104:0/3248012035 <== mon.2 v1:172.21.15.104:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f378c004e10 con 0x7f379c06e540 2024-03-29T11:28:46.039 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.041+0000 7f3783fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:46.039 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.041+0000 7f37997fa640 4 set_mon_vals no callback set 2024-03-29T11:28:46.039 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.041+0000 7f3783fff640 1 -- 172.21.15.104:0/3248012035 <== mon.2 v1:172.21.15.104:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f378c005220 con 0x7f379c06e540 2024-03-29T11:28:46.039 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.041+0000 7f3783fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:46.039 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.041+0000 7f3783fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:46.039 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.041+0000 7f3783fff640 10 monclient: dump: 2024-03-29T11:28:46.040 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:28:46.040 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:28:46.040 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:46.040 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:46.040 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:46.040 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:28:46.040 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:28:46.040 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:28:46.040 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:28:46.040 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:28:46.040 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.042+0000 7f379bfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:46.040 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.042+0000 7f379bfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:46.040 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.042+0000 7f379bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:28:46.040 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.042+0000 7f379bfff640 1 -- 172.21.15.104:0/3248012035 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f377c005080 con 0x7f379c06e540 2024-03-29T11:28:46.040 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.042+0000 7f3783fff640 1 -- 172.21.15.104:0/3248012035 <== mon.2 v1:172.21.15.104:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f378c005380 con 0x7f379c06e540 2024-03-29T11:28:46.041 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.043+0000 7f379affd640 10 monclient: get_auth_request con 0x7f378806dcf0 auth_method 0 2024-03-29T11:28:46.041 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.043+0000 7f3783fff640 10 client.4592.objecter ms_handle_connect 0x7f378806dcf0 2024-03-29T11:28:46.041 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.043+0000 7f3783fff640 1 -- 172.21.15.104:0/3248012035 <== mon.2 v1:172.21.15.104:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f378c084e20 con 0x7f379c06e540 2024-03-29T11:28:46.041 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.044+0000 7f3783fff640 10 client.4592.objecter ms_dispatch 0x7f379c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:46.041 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.044+0000 7f3783fff640 3 client.4592.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:46.041 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.044+0000 7f3783fff640 3 client.4592.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:46.041 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.044+0000 7f3783fff640 20 client.4592.objecter dump_active .. 0 homeless 2024-03-29T11:28:46.044 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.053+0000 7f3783fff640 1 -- 172.21.15.104:0/3248012035 <== mon.2 v1:172.21.15.104:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f378c050530 con 0x7f379c06e540 2024-03-29T11:28:46.044 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.053+0000 7f3783fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:46.044 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.053+0000 7f3783fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:46.368 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.377+0000 7f379bfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:46.368 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.377+0000 7f379bfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:46.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.377+0000 7f379bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:28:46.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.377+0000 7f379bfff640 1 -- 172.21.15.104:0/3248012035 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f377c005330 con 0x7f379c06e540 2024-03-29T11:28:46.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.377+0000 7f3783fff640 1 -- 172.21.15.104:0/3248012035 <== mon.2 v1:172.21.15.104:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f378c052630 con 0x7f379c06e540 2024-03-29T11:28:46.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.377+0000 7f3783fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:46.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.377+0000 7f3783fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:46.370 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:28:46.370 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:46.371 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.381+0000 7f379bfff640 10 librados: watch_flush enter 2024-03-29T11:28:46.374 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.383+0000 7f379bfff640 10 librados: watch_flush exit 2024-03-29T11:28:46.374 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.383+0000 7f379bfff640 20 client.4592.objecter shutdown clearing up homeless session... 2024-03-29T11:28:46.374 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.383+0000 7f379bfff640 10 client.4592.objecter successfully canceled tick 2024-03-29T11:28:46.375 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.383+0000 7f379bfff640 1 -- 172.21.15.104:0/3248012035 >> v1:172.21.15.104:6816/3341126660 conn(0x7f378806dcf0 legacy=0x7f37880701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:46.375 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.383+0000 7f379bfff640 10 monclient: shutdown 2024-03-29T11:28:46.375 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.383+0000 7f379bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:46.375 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.383+0000 7f379bfff640 1 -- 172.21.15.104:0/3248012035 >> v1:172.21.15.104:6790/0 conn(0x7f379c06e540 legacy=0x7f379c1e5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:46.376 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.384+0000 7f379bfff640 1 -- 172.21.15.104:0/3248012035 shutdown_connections 2024-03-29T11:28:46.376 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.384+0000 7f379bfff640 1 -- 172.21.15.104:0/3248012035 >> 172.21.15.104:0/3248012035 conn(0x7f379c0673e0 msgr2=0x7f379c146320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:46.376 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.385+0000 7f379bfff640 1 -- 172.21.15.104:0/3248012035 shutdown_connections 2024-03-29T11:28:46.376 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.385+0000 7f379bfff640 1 -- 172.21.15.104:0/3248012035 wait complete. 2024-03-29T11:28:46.377 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:46.385+0000 7f379bfff640 1 librados: shutdown 2024-03-29T11:28:46.391 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:53.199 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:53.358 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.367+0000 7f3d826ed640 10 monclient: get_monmap_and_config 2024-03-29T11:28:53.358 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.367+0000 7f3d826ed640 10 monclient: build_initial_monmap 2024-03-29T11:28:53.358 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.367+0000 7f3d826ed640 10 monclient: monmap: 2024-03-29T11:28:53.359 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:28:53.359 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:28:53.359 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:28:53.368792+0000 2024-03-29T11:28:53.359 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:28:53.368792+0000 2024-03-29T11:28:53.359 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:53.359 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:28:53.359 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:28:53.359 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:28:53.359 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:28:53.359 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:28:53.377 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.368+0000 7f3d826ed640 1 Processor -- start 2024-03-29T11:28:53.377 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.369+0000 7f3d826ed640 1 -- start start 2024-03-29T11:28:53.377 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.369+0000 7f3d826ed640 10 monclient: init 2024-03-29T11:28:53.378 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.369+0000 7f3d826ed640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:53.378 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.369+0000 7f3d826ed640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:28:53.378 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.369+0000 7f3d826ed640 10 monclient(hunting): picked mon.noname-b con 0x7f3d7c0b0500 addr v1:172.21.15.104:6790/0 2024-03-29T11:28:53.378 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.369+0000 7f3d826ed640 10 monclient(hunting): picked mon.noname-c con 0x7f3d7c0b08e0 addr v1:172.21.15.119:6789/0 2024-03-29T11:28:53.378 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.369+0000 7f3d826ed640 10 monclient(hunting): picked mon.noname-a con 0x7f3d7c0aeca0 addr v1:172.21.15.104:6789/0 2024-03-29T11:28:53.378 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.369+0000 7f3d826ed640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d7c140620 con 0x7f3d7c0aeca0 2024-03-29T11:28:53.378 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.370+0000 7f3d826ed640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d7c13daf0 con 0x7f3d7c0b08e0 2024-03-29T11:28:53.378 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.370+0000 7f3d826ed640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d7c0ae180 con 0x7f3d7c0b0500 2024-03-29T11:28:53.378 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.370+0000 7f3d826ed640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:53.378 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.370+0000 7f3d826ed640 10 monclient(hunting): authenticate will time out at 1514.621044s 2024-03-29T11:28:53.378 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.370+0000 7f3d7b7fe640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f3d7c0b08e0 0x7f3d7c13d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:55116/0 (socket says 172.21.15.104:55116) 2024-03-29T11:28:53.378 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.370+0000 7f3d7b7fe640 1 -- 172.21.15.104:0/3825509395 learned_addr learned my addr 172.21.15.104:0/3825509395 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:28:53.378 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.370+0000 7f3d7affd640 1 -- 172.21.15.104:0/3825509395 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3640650741 0 0) 0x7f3d7c13daf0 con 0x7f3d7c0b08e0 2024-03-29T11:28:53.379 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.370+0000 7f3d7affd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:53.379 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.370+0000 7f3d7affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:53.379 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.370+0000 7f3d7affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:53.379 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.370+0000 7f3d7affd640 10 monclient(hunting): my global_id is 4543 2024-03-29T11:28:53.379 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.370+0000 7f3d7affd640 1 -- 172.21.15.104:0/3825509395 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d68003520 con 0x7f3d7c0b08e0 2024-03-29T11:28:53.379 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.370+0000 7f3d7affd640 1 -- 172.21.15.104:0/3825509395 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1468811255 0 0) 0x7f3d7c140620 con 0x7f3d7c0aeca0 2024-03-29T11:28:53.379 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.370+0000 7f3d7affd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:53.379 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.370+0000 7f3d7affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:53.379 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.371+0000 7f3d7affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:53.379 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.371+0000 7f3d7affd640 10 monclient(hunting): my global_id is 4557 2024-03-29T11:28:53.379 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.371+0000 7f3d7affd640 1 -- 172.21.15.104:0/3825509395 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d7c13daf0 con 0x7f3d7c0aeca0 2024-03-29T11:28:53.379 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.371+0000 7f3d7affd640 1 -- 172.21.15.104:0/3825509395 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2005577757 0 0) 0x7f3d68003520 con 0x7f3d7c0b08e0 2024-03-29T11:28:53.379 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.371+0000 7f3d7affd640 1 -- 172.21.15.104:0/3825509395 >> v1:172.21.15.104:6790/0 conn(0x7f3d7c0b0500 legacy=0x7f3d7c13ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:53.379 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.371+0000 7f3d7affd640 1 -- 172.21.15.104:0/3825509395 >> v1:172.21.15.104:6789/0 conn(0x7f3d7c0aeca0 legacy=0x7f3d7c14b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:53.380 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.371+0000 7f3d7affd640 10 monclient: _finish_hunting 0 2024-03-29T11:28:53.380 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.371+0000 7f3d7affd640 1 monclient: found mon.noname-c 2024-03-29T11:28:53.380 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.371+0000 7f3d7affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:53.380 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.371+0000 7f3d7affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.119:6789/0 2024-03-29T11:28:53.380 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.371+0000 7f3d7affd640 1 -- 172.21.15.104:0/3825509395 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d7c0accf0 con 0x7f3d7c0b08e0 2024-03-29T11:28:53.380 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.371+0000 7f3d7affd640 10 monclient: _finish_auth 0 2024-03-29T11:28:53.380 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.371+0000 7f3d7affd640 10 monclient: _check_auth_tickets 2024-03-29T11:28:53.380 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.371+0000 7f3d7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:53.380 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.371+0000 7f3d826ed640 5 monclient: authenticate success, global_id 4543 2024-03-29T11:28:53.380 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.371+0000 7f3d826ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:53.380 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.371+0000 7f3d7affd640 1 -- 172.21.15.104:0/3825509395 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f3d6c001690 con 0x7f3d7c0b08e0 2024-03-29T11:28:53.380 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.371+0000 7f3d7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:53.380 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.371+0000 7f3d7affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:53.380 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.371+0000 7f3d7affd640 10 monclient: dump: 2024-03-29T11:28:53.380 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:28:53.381 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:28:53.381 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:53.381 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:53.381 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:53.381 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:28:53.381 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:28:53.381 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:28:53.381 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:28:53.381 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:28:53.381 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.371+0000 7f3d826ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:53.381 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.371+0000 7f3d7affd640 1 -- 172.21.15.104:0/3825509395 <== mon.1 v1:172.21.15.119:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d6c0018b0 con 0x7f3d7c0b08e0 2024-03-29T11:28:53.381 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.371+0000 7f3d7affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:53.381 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.371+0000 7f3d826ed640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:53.381 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.371+0000 7f3d826ed640 4 set_mon_vals no callback set 2024-03-29T11:28:53.381 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.372+0000 7f3d7affd640 1 -- 172.21.15.104:0/3825509395 <== mon.1 v1:172.21.15.119:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f3d6c001c40 con 0x7f3d7c0b08e0 2024-03-29T11:28:53.382 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.372+0000 7f3d7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:53.382 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.372+0000 7f3d7affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:53.382 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.372+0000 7f3d826ed640 10 monclient: shutdown 2024-03-29T11:28:53.382 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.372+0000 7f3d7affd640 10 monclient: dump: 2024-03-29T11:28:53.382 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:28:53.382 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:28:53.382 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:53.382 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:53.382 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:53.382 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:28:53.382 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:28:53.382 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:28:53.382 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:28:53.382 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:28:53.382 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.372+0000 7f3d826ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:53.383 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.372+0000 7f3d826ed640 1 -- 172.21.15.104:0/3825509395 >> v1:172.21.15.119:6789/0 conn(0x7f3d7c0b08e0 legacy=0x7f3d7c13d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:53.383 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.372+0000 7f3d826ed640 1 -- 172.21.15.104:0/3825509395 shutdown_connections 2024-03-29T11:28:53.383 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.372+0000 7f3d826ed640 1 -- 172.21.15.104:0/3825509395 >> 172.21.15.104:0/3825509395 conn(0x7f3d7c06b660 msgr2=0x7f3d7c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:53.383 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.372+0000 7f3d826ed640 1 -- 172.21.15.104:0/3825509395 shutdown_connections 2024-03-29T11:28:53.383 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.372+0000 7f3d826ed640 1 -- 172.21.15.104:0/3825509395 wait complete. 2024-03-29T11:28:53.383 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.373+0000 7f3d826ed640 10 monclient: build_initial_monmap 2024-03-29T11:28:53.383 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.373+0000 7f3d826ed640 10 monclient: monmap: 2024-03-29T11:28:53.383 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:28:53.383 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:53.383 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:28:53.383 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:28:53.383 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:53.383 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:28:53.383 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:28:53.384 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:28:53.384 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:28:53.384 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:28:53.384 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.374+0000 7f3d826ed640 1 librados: starting msgr at 2024-03-29T11:28:53.384 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.374+0000 7f3d826ed640 1 librados: starting objecter 2024-03-29T11:28:53.384 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.374+0000 7f3d826ed640 1 Processor -- start 2024-03-29T11:28:53.384 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.374+0000 7f3d826ed640 1 -- start start 2024-03-29T11:28:53.384 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.374+0000 7f3d826ed640 1 librados: setting wanted keys 2024-03-29T11:28:53.384 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.374+0000 7f3d826ed640 1 librados: calling monclient init 2024-03-29T11:28:53.384 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.374+0000 7f3d826ed640 10 monclient: init 2024-03-29T11:28:53.384 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.375+0000 7f3d826ed640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:53.384 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.375+0000 7f3d826ed640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:28:53.384 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.375+0000 7f3d826ed640 10 monclient(hunting): picked mon.noname-c con 0x7f3d7c0aeca0 addr v1:172.21.15.104:6790/0 2024-03-29T11:28:53.384 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.375+0000 7f3d7bfff640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f3d7c0aeca0 0x7f3d7c070320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:47534/0 (socket says 172.21.15.104:47534) 2024-03-29T11:28:53.384 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.376+0000 7f3d7bfff640 1 -- 172.21.15.104:0/1936681862 learned_addr learned my addr 172.21.15.104:0/1936681862 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:28:53.384 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.376+0000 7f3d826ed640 10 monclient(hunting): picked mon.noname-a con 0x7f3d7c0b0500 addr v1:172.21.15.104:6789/0 2024-03-29T11:28:53.385 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.376+0000 7f3d78ff9640 10 client.?.objecter ms_handle_connect 0x7f3d7c0aeca0 2024-03-29T11:28:53.385 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.376+0000 7f3d78ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:53.385 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.376+0000 7f3d78ff9640 10 client.?.objecter ms_handle_connect 0x7f3d7c0b0500 2024-03-29T11:28:53.385 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.376+0000 7f3d78ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:53.385 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.376+0000 7f3d826ed640 10 monclient(hunting): picked mon.noname-b con 0x7f3d7c070a30 addr v1:172.21.15.119:6789/0 2024-03-29T11:28:53.385 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.376+0000 7f3d826ed640 1 -- 172.21.15.104:0/1936681862 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d7c06f5e0 con 0x7f3d7c0b0500 2024-03-29T11:28:53.385 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.376+0000 7f3d78ff9640 10 client.?.objecter ms_handle_connect 0x7f3d7c070a30 2024-03-29T11:28:53.385 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.376+0000 7f3d78ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:53.385 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.376+0000 7f3d826ed640 1 -- 172.21.15.104:0/1936681862 --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d7c138230 con 0x7f3d7c070a30 2024-03-29T11:28:53.385 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.376+0000 7f3d826ed640 1 -- 172.21.15.104:0/1936681862 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d7c06f0c0 con 0x7f3d7c0aeca0 2024-03-29T11:28:53.385 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.376+0000 7f3d826ed640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:53.385 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.376+0000 7f3d826ed640 10 monclient(hunting): authenticate will time out at 1514.627557s 2024-03-29T11:28:53.385 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.377+0000 7f3d78ff9640 1 -- 172.21.15.104:0/1936681862 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3654859719 0 0) 0x7f3d7c138230 con 0x7f3d7c070a30 2024-03-29T11:28:53.385 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.377+0000 7f3d78ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:28:53.386 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.377+0000 7f3d78ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:53.386 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.377+0000 7f3d78ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:53.386 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.377+0000 7f3d78ff9640 10 monclient(hunting): my global_id is 4546 2024-03-29T11:28:53.386 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.377+0000 7f3d78ff9640 1 -- 172.21.15.104:0/1936681862 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d5c004eb0 con 0x7f3d7c070a30 2024-03-29T11:28:53.386 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.377+0000 7f3d78ff9640 1 -- 172.21.15.104:0/1936681862 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1405517578 0 0) 0x7f3d7c06f0c0 con 0x7f3d7c0aeca0 2024-03-29T11:28:53.386 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.377+0000 7f3d78ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:28:53.386 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.377+0000 7f3d78ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:53.386 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.377+0000 7f3d78ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:53.386 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.377+0000 7f3d78ff9640 10 monclient(hunting): my global_id is 4598 2024-03-29T11:28:53.386 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.377+0000 7f3d78ff9640 1 -- 172.21.15.104:0/1936681862 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d7c138230 con 0x7f3d7c0aeca0 2024-03-29T11:28:53.386 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.377+0000 7f3d78ff9640 1 -- 172.21.15.104:0/1936681862 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 651478980 0 0) 0x7f3d7c06f5e0 con 0x7f3d7c0b0500 2024-03-29T11:28:53.386 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.377+0000 7f3d78ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:28:53.386 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.377+0000 7f3d78ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:53.386 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.377+0000 7f3d78ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:53.386 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.377+0000 7f3d78ff9640 10 monclient(hunting): my global_id is 4560 2024-03-29T11:28:53.387 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.377+0000 7f3d78ff9640 1 -- 172.21.15.104:0/1936681862 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d7c06f0c0 con 0x7f3d7c0b0500 2024-03-29T11:28:53.387 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.378+0000 7f3d78ff9640 1 -- 172.21.15.104:0/1936681862 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 567402194 0 0) 0x7f3d5c004eb0 con 0x7f3d7c070a30 2024-03-29T11:28:53.387 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.378+0000 7f3d78ff9640 1 -- 172.21.15.104:0/1936681862 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d7c06f5e0 con 0x7f3d7c070a30 2024-03-29T11:28:53.387 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.378+0000 7f3d78ff9640 1 -- 172.21.15.104:0/1936681862 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f3d74001cb0 con 0x7f3d7c070a30 2024-03-29T11:28:53.387 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.378+0000 7f3d78ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:53.387 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.378+0000 7f3d78ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:53.387 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.378+0000 7f3d78ff9640 10 monclient(hunting): dump: 2024-03-29T11:28:53.387 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:28:53.387 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:28:53.387 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:53.387 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:53.387 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:53.387 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:28:53.387 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:28:53.387 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:28:53.388 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:28:53.388 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:28:53.388 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.378+0000 7f3d78ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:53.388 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.378+0000 7f3d78ff9640 1 -- 172.21.15.104:0/1936681862 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2374924169 0 0) 0x7f3d7c138230 con 0x7f3d7c0aeca0 2024-03-29T11:28:53.388 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.378+0000 7f3d78ff9640 1 -- 172.21.15.104:0/1936681862 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d5c004eb0 con 0x7f3d7c0aeca0 2024-03-29T11:28:53.388 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.378+0000 7f3d78ff9640 1 -- 172.21.15.104:0/1936681862 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f3d640019b0 con 0x7f3d7c0aeca0 2024-03-29T11:28:53.388 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.378+0000 7f3d78ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:53.388 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.378+0000 7f3d78ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:53.388 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.378+0000 7f3d78ff9640 10 monclient(hunting): dump: 2024-03-29T11:28:53.388 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:28:53.388 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:28:53.388 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:53.388 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:53.389 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:53.389 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:28:53.389 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:28:53.389 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:28:53.389 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:28:53.389 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:28:53.389 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.378+0000 7f3d78ff9640 1 -- 172.21.15.104:0/1936681862 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 735061907 0 0) 0x7f3d7c06f5e0 con 0x7f3d7c070a30 2024-03-29T11:28:53.389 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.378+0000 7f3d78ff9640 1 -- 172.21.15.104:0/1936681862 >> v1:172.21.15.104:6790/0 conn(0x7f3d7c0aeca0 legacy=0x7f3d7c070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:53.389 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.378+0000 7f3d78ff9640 1 -- 172.21.15.104:0/1936681862 >> v1:172.21.15.104:6789/0 conn(0x7f3d7c0b0500 legacy=0x7f3d7c138930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:53.389 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.378+0000 7f3d78ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:28:53.389 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.378+0000 7f3d78ff9640 1 monclient: found mon.b 2024-03-29T11:28:53.389 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.378+0000 7f3d78ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:53.389 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.378+0000 7f3d78ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:28:53.389 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.378+0000 7f3d78ff9640 1 -- 172.21.15.104:0/1936681862 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d7c06ede0 con 0x7f3d7c070a30 2024-03-29T11:28:53.390 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.378+0000 7f3d78ff9640 10 monclient: _finish_auth 0 2024-03-29T11:28:53.390 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.378+0000 7f3d78ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:28:53.390 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.378+0000 7f3d78ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:53.390 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.379+0000 7f3d826ed640 5 monclient: authenticate success, global_id 4546 2024-03-29T11:28:53.390 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.379+0000 7f3d826ed640 10 monclient: _renew_subs 2024-03-29T11:28:53.390 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.379+0000 7f3d826ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:28:53.390 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.379+0000 7f3d826ed640 1 -- 172.21.15.104:0/1936681862 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d7c1e3620 con 0x7f3d7c070a30 2024-03-29T11:28:53.390 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.379+0000 7f3d826ed640 10 client.4546.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:53.390 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.379+0000 7f3d826ed640 10 monclient: _renew_subs 2024-03-29T11:28:53.390 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.379+0000 7f3d826ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:28:53.390 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.379+0000 7f3d826ed640 1 -- 172.21.15.104:0/1936681862 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d7c1e3990 con 0x7f3d7c070a30 2024-03-29T11:28:53.390 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.379+0000 7f3d826ed640 1 librados: init done 2024-03-29T11:28:53.390 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.379+0000 7f3d78ff9640 1 -- 172.21.15.104:0/1936681862 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d74003990 con 0x7f3d7c070a30 2024-03-29T11:28:53.390 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.379+0000 7f3d78ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:53.391 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.379+0000 7f3d78ff9640 1 -- 172.21.15.104:0/1936681862 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f3d74003ca0 con 0x7f3d7c070a30 2024-03-29T11:28:53.391 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.379+0000 7f3d78ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:53.391 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.379+0000 7f3d78ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:53.391 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.379+0000 7f3d78ff9640 10 monclient: dump: 2024-03-29T11:28:53.391 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:28:53.391 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:28:53.391 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:53.391 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:28:53.391 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:53.391 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:28:53.391 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:28:53.391 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:28:53.391 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:28:53.391 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:28:53.391 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.379+0000 7f3d7a7fc640 4 set_mon_vals no callback set 2024-03-29T11:28:53.391 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.381+0000 7f3d78ff9640 1 -- 172.21.15.104:0/1936681862 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f3d74019c70 con 0x7f3d7c070a30 2024-03-29T11:28:53.392 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.382+0000 7f3d78ff9640 1 -- 172.21.15.104:0/1936681862 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f3d74083b40 con 0x7f3d7c070a30 2024-03-29T11:28:53.392 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.382+0000 7f3d78ff9640 10 client.4546.objecter ms_dispatch 0x7f3d7c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:53.392 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.382+0000 7f3d78ff9640 3 client.4546.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:53.392 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.382+0000 7f3d78ff9640 3 client.4546.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:53.392 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.382+0000 7f3d78ff9640 20 client.4546.objecter dump_active .. 0 homeless 2024-03-29T11:28:53.392 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.382+0000 7f3d826ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:53.392 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.382+0000 7f3d826ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:53.392 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.382+0000 7f3d826ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:28:53.392 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.382+0000 7f3d826ed640 1 -- 172.21.15.104:0/1936681862 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d60005080 con 0x7f3d7c070a30 2024-03-29T11:28:53.392 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.382+0000 7f3d7bfff640 10 monclient: get_auth_request con 0x7f3d5c06df90 auth_method 0 2024-03-29T11:28:53.393 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.383+0000 7f3d78ff9640 10 client.4546.objecter ms_handle_connect 0x7f3d5c06df90 2024-03-29T11:28:53.393 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.389+0000 7f3d78ff9640 1 -- 172.21.15.104:0/1936681862 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3d7404f250 con 0x7f3d7c070a30 2024-03-29T11:28:53.393 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.389+0000 7f3d78ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:53.393 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.389+0000 7f3d78ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:53.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.749+0000 7f3d826ed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:53.741 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.749+0000 7f3d826ed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:53.741 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.749+0000 7f3d826ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:28:53.741 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.749+0000 7f3d826ed640 1 -- 172.21.15.104:0/1936681862 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d60005330 con 0x7f3d7c070a30 2024-03-29T11:28:53.743 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:28:53.743 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:53.743 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.752+0000 7f3d78ff9640 1 -- 172.21.15.104:0/1936681862 <== mon.1 v1:172.21.15.119:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f3d74051350 con 0x7f3d7c070a30 2024-03-29T11:28:53.743 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.752+0000 7f3d78ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:53.744 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.752+0000 7f3d78ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:53.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.756+0000 7f3d826ed640 10 librados: watch_flush enter 2024-03-29T11:28:53.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.756+0000 7f3d826ed640 10 librados: watch_flush exit 2024-03-29T11:28:53.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.756+0000 7f3d826ed640 20 client.4546.objecter shutdown clearing up homeless session... 2024-03-29T11:28:53.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.756+0000 7f3d826ed640 10 client.4546.objecter successfully canceled tick 2024-03-29T11:28:53.748 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.756+0000 7f3d826ed640 1 -- 172.21.15.104:0/1936681862 >> v1:172.21.15.104:6816/3341126660 conn(0x7f3d5c06df90 legacy=0x7f3d5c070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:53.748 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.756+0000 7f3d826ed640 10 monclient: shutdown 2024-03-29T11:28:53.748 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.756+0000 7f3d826ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:53.748 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.757+0000 7f3d826ed640 1 -- 172.21.15.104:0/1936681862 >> v1:172.21.15.119:6789/0 conn(0x7f3d7c070a30 legacy=0x7f3d7c139040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:53.748 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.757+0000 7f3d826ed640 1 -- 172.21.15.104:0/1936681862 shutdown_connections 2024-03-29T11:28:53.748 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.757+0000 7f3d826ed640 1 -- 172.21.15.104:0/1936681862 >> 172.21.15.104:0/1936681862 conn(0x7f3d7c06b660 msgr2=0x7f3d7c0a9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:53.748 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.757+0000 7f3d826ed640 1 -- 172.21.15.104:0/1936681862 shutdown_connections 2024-03-29T11:28:53.748 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.757+0000 7f3d826ed640 1 -- 172.21.15.104:0/1936681862 wait complete. 2024-03-29T11:28:53.748 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:28:53.757+0000 7f3d826ed640 1 librados: shutdown 2024-03-29T11:28:53.764 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:00.572 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:00.724 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.734+0000 7fa7ddce6640 10 monclient: get_monmap_and_config 2024-03-29T11:29:00.725 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.734+0000 7fa7ddce6640 10 monclient: build_initial_monmap 2024-03-29T11:29:00.725 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.734+0000 7fa7ddce6640 10 monclient: monmap: 2024-03-29T11:29:00.725 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:29:00.725 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:29:00.725 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:29:00.735087+0000 2024-03-29T11:29:00.725 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:29:00.735087+0000 2024-03-29T11:29:00.725 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:00.725 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:00.725 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:29:00.725 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:29:00.726 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:29:00.726 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:00.727 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.735+0000 7fa7ddce6640 1 Processor -- start 2024-03-29T11:29:00.727 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.735+0000 7fa7ddce6640 1 -- start start 2024-03-29T11:29:00.727 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.735+0000 7fa7ddce6640 10 monclient: init 2024-03-29T11:29:00.727 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.736+0000 7fa7ddce6640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:00.727 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.736+0000 7fa7ddce6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:29:00.727 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.736+0000 7fa7ddce6640 10 monclient(hunting): picked mon.noname-a con 0x7fa7d00389e0 addr v1:172.21.15.104:6789/0 2024-03-29T11:29:00.727 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.736+0000 7fa7ddce6640 10 monclient(hunting): picked mon.noname-c con 0x7fa7d0038dc0 addr v1:172.21.15.119:6789/0 2024-03-29T11:29:00.727 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.736+0000 7fa7ddce6640 10 monclient(hunting): picked mon.noname-b con 0x7fa7d0030660 addr v1:172.21.15.104:6790/0 2024-03-29T11:29:00.728 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.736+0000 7fa7ddce6640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7d00c54c0 con 0x7fa7d00389e0 2024-03-29T11:29:00.728 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.736+0000 7fa7ddce6640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7d0036430 con 0x7fa7d0038dc0 2024-03-29T11:29:00.728 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.736+0000 7fa7ddce6640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7d0039840 con 0x7fa7d0030660 2024-03-29T11:29:00.728 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.736+0000 7fa7ddce6640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:00.728 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.736+0000 7fa7ddce6640 10 monclient(hunting): authenticate will time out at 1521.987711s 2024-03-29T11:29:00.728 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.737+0000 7fa7d6ffd640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fa7d0038dc0 0x7fa7d00cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:55554/0 (socket says 172.21.15.104:55554) 2024-03-29T11:29:00.728 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.737+0000 7fa7d6ffd640 1 -- 172.21.15.104:0/3262146742 learned_addr learned my addr 172.21.15.104:0/3262146742 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:29:00.729 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.737+0000 7fa7d67fc640 1 -- 172.21.15.104:0/3262146742 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2280904079 0 0) 0x7fa7d0036430 con 0x7fa7d0038dc0 2024-03-29T11:29:00.729 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.737+0000 7fa7d67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:29:00.729 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.737+0000 7fa7d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:00.729 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.737+0000 7fa7d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:00.729 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.737+0000 7fa7d67fc640 10 monclient(hunting): my global_id is 4549 2024-03-29T11:29:00.729 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.737+0000 7fa7d67fc640 1 -- 172.21.15.104:0/3262146742 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7c8003520 con 0x7fa7d0038dc0 2024-03-29T11:29:00.729 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.737+0000 7fa7d67fc640 1 -- 172.21.15.104:0/3262146742 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 39739620 0 0) 0x7fa7d0039840 con 0x7fa7d0030660 2024-03-29T11:29:00.729 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.738+0000 7fa7d67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:29:00.730 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.738+0000 7fa7d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:00.730 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.738+0000 7fa7d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:00.730 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.738+0000 7fa7d67fc640 10 monclient(hunting): my global_id is 4601 2024-03-29T11:29:00.730 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.738+0000 7fa7d67fc640 1 -- 172.21.15.104:0/3262146742 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7d0036430 con 0x7fa7d0030660 2024-03-29T11:29:00.730 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.738+0000 7fa7d67fc640 1 -- 172.21.15.104:0/3262146742 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2799847986 0 0) 0x7fa7c8003520 con 0x7fa7d0038dc0 2024-03-29T11:29:00.730 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.738+0000 7fa7d67fc640 1 -- 172.21.15.104:0/3262146742 >> v1:172.21.15.104:6790/0 conn(0x7fa7d0030660 legacy=0x7fa7d00d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:00.730 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.738+0000 7fa7d67fc640 1 -- 172.21.15.104:0/3262146742 >> v1:172.21.15.104:6789/0 conn(0x7fa7d00389e0 legacy=0x7fa7d0033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:00.730 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.738+0000 7fa7d67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:29:00.730 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.738+0000 7fa7d67fc640 1 monclient: found mon.noname-c 2024-03-29T11:29:00.730 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.738+0000 7fa7d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:00.731 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.738+0000 7fa7d67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.119:6789/0 2024-03-29T11:29:00.731 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.738+0000 7fa7d67fc640 1 -- 172.21.15.104:0/3262146742 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7d0035690 con 0x7fa7d0038dc0 2024-03-29T11:29:00.731 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.738+0000 7fa7d67fc640 10 monclient: _finish_auth 0 2024-03-29T11:29:00.731 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.738+0000 7fa7d67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:29:00.731 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.738+0000 7fa7d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:00.731 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.738+0000 7fa7d67fc640 1 -- 172.21.15.104:0/3262146742 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fa7cc001690 con 0x7fa7d0038dc0 2024-03-29T11:29:00.731 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.738+0000 7fa7d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:00.731 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.738+0000 7fa7d67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:00.731 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.738+0000 7fa7d67fc640 10 monclient: dump: 2024-03-29T11:29:00.731 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:29:00.732 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:29:00.732 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:00.732 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:00.732 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:00.732 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:00.732 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:29:00.732 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:29:00.732 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:29:00.732 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:00.732 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.738+0000 7fa7ddce6640 5 monclient: authenticate success, global_id 4549 2024-03-29T11:29:00.732 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.738+0000 7fa7ddce6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:00.732 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.738+0000 7fa7d67fc640 1 -- 172.21.15.104:0/3262146742 <== mon.1 v1:172.21.15.119:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7cc001860 con 0x7fa7d0038dc0 2024-03-29T11:29:00.732 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.738+0000 7fa7d67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:00.732 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.738+0000 7fa7d67fc640 1 -- 172.21.15.104:0/3262146742 <== mon.1 v1:172.21.15.119:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fa7cc001c70 con 0x7fa7d0038dc0 2024-03-29T11:29:00.732 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.738+0000 7fa7d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:00.733 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.738+0000 7fa7d67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:00.733 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.738+0000 7fa7d67fc640 10 monclient: dump: 2024-03-29T11:29:00.733 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:29:00.733 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:29:00.733 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:00.733 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:00.733 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:00.733 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:00.733 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:29:00.733 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:29:00.733 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:29:00.733 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:00.733 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.738+0000 7fa7ddce6640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:00.733 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.738+0000 7fa7ddce6640 4 set_mon_vals no callback set 2024-03-29T11:29:00.743 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.739+0000 7fa7ddce6640 10 monclient: shutdown 2024-03-29T11:29:00.743 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.739+0000 7fa7ddce6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:00.743 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.739+0000 7fa7ddce6640 1 -- 172.21.15.104:0/3262146742 >> v1:172.21.15.119:6789/0 conn(0x7fa7d0038dc0 legacy=0x7fa7d00cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:00.743 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.739+0000 7fa7ddce6640 1 -- 172.21.15.104:0/3262146742 shutdown_connections 2024-03-29T11:29:00.743 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.739+0000 7fa7ddce6640 1 -- 172.21.15.104:0/3262146742 >> 172.21.15.104:0/3262146742 conn(0x7fa7d0005c00 msgr2=0x7fa7d0006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:00.743 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.739+0000 7fa7ddce6640 1 -- 172.21.15.104:0/3262146742 shutdown_connections 2024-03-29T11:29:00.743 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.739+0000 7fa7ddce6640 1 -- 172.21.15.104:0/3262146742 wait complete. 2024-03-29T11:29:00.743 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.741+0000 7fa7ddce6640 10 monclient: build_initial_monmap 2024-03-29T11:29:00.743 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.741+0000 7fa7ddce6640 10 monclient: monmap: 2024-03-29T11:29:00.743 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:29:00.744 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:00.744 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:29:00.744 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:29:00.744 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:00.744 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:00.744 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:29:00.744 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:29:00.744 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:29:00.744 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:00.744 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.741+0000 7fa7ddce6640 1 librados: starting msgr at 2024-03-29T11:29:00.744 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.741+0000 7fa7ddce6640 1 librados: starting objecter 2024-03-29T11:29:00.744 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.741+0000 7fa7ddce6640 1 Processor -- start 2024-03-29T11:29:00.744 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.742+0000 7fa7ddce6640 1 -- start start 2024-03-29T11:29:00.744 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.742+0000 7fa7ddce6640 1 librados: setting wanted keys 2024-03-29T11:29:00.744 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.742+0000 7fa7ddce6640 1 librados: calling monclient init 2024-03-29T11:29:00.745 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.742+0000 7fa7ddce6640 10 monclient: init 2024-03-29T11:29:00.745 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.743+0000 7fa7ddce6640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:00.745 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.743+0000 7fa7ddce6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:29:00.745 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.743+0000 7fa7ddce6640 10 monclient(hunting): picked mon.noname-a con 0x7fa7d0030660 addr v1:172.21.15.104:6789/0 2024-03-29T11:29:00.745 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.744+0000 7fa7ddce6640 10 monclient(hunting): picked mon.noname-c con 0x7fa7d00389e0 addr v1:172.21.15.104:6790/0 2024-03-29T11:29:00.745 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.744+0000 7fa7ddce6640 10 monclient(hunting): picked mon.noname-b con 0x7fa7d00c7550 addr v1:172.21.15.119:6789/0 2024-03-29T11:29:00.745 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.744+0000 7fa7ddce6640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7d0009dd0 con 0x7fa7d0030660 2024-03-29T11:29:00.745 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.744+0000 7fa7ddce6640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7d002f5f0 con 0x7fa7d00c7550 2024-03-29T11:29:00.745 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.744+0000 7fa7d6ffd640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7fa7d00389e0 0x7fa7d015bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:37198/0 (socket says 172.21.15.104:37198) 2024-03-29T11:29:00.745 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.744+0000 7fa7d6ffd640 1 -- 172.21.15.104:0/29993141 learned_addr learned my addr 172.21.15.104:0/29993141 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:29:00.745 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.744+0000 7fa7ddce6640 1 -- 172.21.15.104:0/29993141 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7d015c280 con 0x7fa7d00389e0 2024-03-29T11:29:00.745 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.744+0000 7fa7ddce6640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:00.745 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.744+0000 7fa7ddce6640 10 monclient(hunting): authenticate will time out at 1521.995534s 2024-03-29T11:29:00.745 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.744+0000 7fa7c3fff640 10 client.?.objecter ms_handle_connect 0x7fa7d0030660 2024-03-29T11:29:00.745 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.744+0000 7fa7c3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:00.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.744+0000 7fa7c3fff640 10 client.?.objecter ms_handle_connect 0x7fa7d00c7550 2024-03-29T11:29:00.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.744+0000 7fa7c3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:00.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.744+0000 7fa7c3fff640 10 client.?.objecter ms_handle_connect 0x7fa7d00389e0 2024-03-29T11:29:00.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.744+0000 7fa7c3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:00.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.744+0000 7fa7c3fff640 1 -- 172.21.15.104:0/29993141 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2308275885 0 0) 0x7fa7d0009dd0 con 0x7fa7d0030660 2024-03-29T11:29:00.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.744+0000 7fa7c3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:29:00.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.744+0000 7fa7c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:00.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.744+0000 7fa7c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:00.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.744+0000 7fa7c3fff640 10 monclient(hunting): my global_id is 4563 2024-03-29T11:29:00.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.744+0000 7fa7c3fff640 1 -- 172.21.15.104:0/29993141 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7bc004eb0 con 0x7fa7d0030660 2024-03-29T11:29:00.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.744+0000 7fa7c3fff640 1 -- 172.21.15.104:0/29993141 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2824730982 0 0) 0x7fa7d015c280 con 0x7fa7d00389e0 2024-03-29T11:29:00.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.745+0000 7fa7c3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:29:00.747 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.745+0000 7fa7c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:00.747 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.745+0000 7fa7c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:00.747 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.745+0000 7fa7c3fff640 10 monclient(hunting): my global_id is 4604 2024-03-29T11:29:00.747 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.745+0000 7fa7c3fff640 1 -- 172.21.15.104:0/29993141 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7d0009dd0 con 0x7fa7d00389e0 2024-03-29T11:29:00.747 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.745+0000 7fa7c3fff640 1 -- 172.21.15.104:0/29993141 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4113391049 0 0) 0x7fa7d002f5f0 con 0x7fa7d00c7550 2024-03-29T11:29:00.747 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.745+0000 7fa7c3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:29:00.747 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.745+0000 7fa7c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:00.747 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.745+0000 7fa7c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:00.747 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.745+0000 7fa7c3fff640 10 monclient(hunting): my global_id is 4552 2024-03-29T11:29:00.747 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.745+0000 7fa7c3fff640 1 -- 172.21.15.104:0/29993141 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7d015c280 con 0x7fa7d00c7550 2024-03-29T11:29:00.747 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.745+0000 7fa7c3fff640 1 -- 172.21.15.104:0/29993141 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1609147247 0 0) 0x7fa7d015c280 con 0x7fa7d00c7550 2024-03-29T11:29:00.747 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.745+0000 7fa7c3fff640 1 -- 172.21.15.104:0/29993141 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7d002f5f0 con 0x7fa7d00c7550 2024-03-29T11:29:00.748 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.745+0000 7fa7c3fff640 1 -- 172.21.15.104:0/29993141 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fa7d804fdd0 con 0x7fa7d00c7550 2024-03-29T11:29:00.748 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.745+0000 7fa7c3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:00.748 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.745+0000 7fa7c3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:00.748 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.745+0000 7fa7c3fff640 10 monclient(hunting): dump: 2024-03-29T11:29:00.748 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:29:00.748 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:29:00.748 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:00.748 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:00.748 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:00.748 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:00.748 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:29:00.748 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:29:00.748 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:29:00.748 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:00.748 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.745+0000 7fa7c3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:00.748 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.745+0000 7fa7c3fff640 1 -- 172.21.15.104:0/29993141 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3665275682 0 0) 0x7fa7bc004eb0 con 0x7fa7d0030660 2024-03-29T11:29:00.749 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.745+0000 7fa7c3fff640 1 -- 172.21.15.104:0/29993141 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7d015c280 con 0x7fa7d0030660 2024-03-29T11:29:00.749 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.745+0000 7fa7c3fff640 1 -- 172.21.15.104:0/29993141 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fa7c4001930 con 0x7fa7d0030660 2024-03-29T11:29:00.749 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.745+0000 7fa7c3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:00.749 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.745+0000 7fa7c3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:00.749 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.745+0000 7fa7c3fff640 10 monclient(hunting): dump: 2024-03-29T11:29:00.749 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:29:00.749 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:29:00.749 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:00.749 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:00.750 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:00.750 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:00.750 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:29:00.750 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:29:00.750 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:29:00.750 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:00.750 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.746+0000 7fa7c3fff640 1 -- 172.21.15.104:0/29993141 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3610253974 0 0) 0x7fa7d0009dd0 con 0x7fa7d00389e0 2024-03-29T11:29:00.750 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.746+0000 7fa7c3fff640 1 -- 172.21.15.104:0/29993141 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7bc004eb0 con 0x7fa7d00389e0 2024-03-29T11:29:00.750 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.746+0000 7fa7c3fff640 1 -- 172.21.15.104:0/29993141 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fa7cc001b70 con 0x7fa7d00389e0 2024-03-29T11:29:00.750 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.746+0000 7fa7c3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:00.750 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.746+0000 7fa7c3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:00.750 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.746+0000 7fa7c3fff640 10 monclient(hunting): dump: 2024-03-29T11:29:00.750 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:29:00.751 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:29:00.751 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:00.751 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:00.751 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:00.751 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:00.751 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:29:00.751 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:29:00.751 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:29:00.751 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:00.751 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.746+0000 7fa7c3fff640 1 -- 172.21.15.104:0/29993141 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3435571749 0 0) 0x7fa7d002f5f0 con 0x7fa7d00c7550 2024-03-29T11:29:00.752 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.746+0000 7fa7c3fff640 1 -- 172.21.15.104:0/29993141 >> v1:172.21.15.104:6790/0 conn(0x7fa7d00389e0 legacy=0x7fa7d015bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:00.752 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.746+0000 7fa7c3fff640 1 -- 172.21.15.104:0/29993141 >> v1:172.21.15.104:6789/0 conn(0x7fa7d0030660 legacy=0x7fa7d002ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:00.752 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.746+0000 7fa7c3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:29:00.752 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.746+0000 7fa7c3fff640 1 monclient: found mon.b 2024-03-29T11:29:00.752 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.746+0000 7fa7c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:00.752 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.746+0000 7fa7c3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:29:00.752 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.746+0000 7fa7c3fff640 1 -- 172.21.15.104:0/29993141 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7d015c410 con 0x7fa7d00c7550 2024-03-29T11:29:00.752 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.746+0000 7fa7c3fff640 10 monclient: _finish_auth 0 2024-03-29T11:29:00.752 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.746+0000 7fa7c3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:29:00.752 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.746+0000 7fa7c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:00.752 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.746+0000 7fa7ddce6640 5 monclient: authenticate success, global_id 4552 2024-03-29T11:29:00.752 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.746+0000 7fa7ddce6640 10 monclient: _renew_subs 2024-03-29T11:29:00.752 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.746+0000 7fa7ddce6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:29:00.752 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.746+0000 7fa7ddce6640 1 -- 172.21.15.104:0/29993141 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7d015c6c0 con 0x7fa7d00c7550 2024-03-29T11:29:00.753 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.746+0000 7fa7ddce6640 10 client.4552.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:00.753 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.746+0000 7fa7ddce6640 10 monclient: _renew_subs 2024-03-29T11:29:00.753 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.746+0000 7fa7ddce6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:29:00.753 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.746+0000 7fa7ddce6640 1 -- 172.21.15.104:0/29993141 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7d0164a80 con 0x7fa7d00c7550 2024-03-29T11:29:00.753 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.746+0000 7fa7c3fff640 1 -- 172.21.15.104:0/29993141 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7d804f400 con 0x7fa7d00c7550 2024-03-29T11:29:00.753 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.746+0000 7fa7ddce6640 1 librados: init done 2024-03-29T11:29:00.753 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.746+0000 7fa7c3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:00.753 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.746+0000 7fa7c3fff640 1 -- 172.21.15.104:0/29993141 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fa7d8075f10 con 0x7fa7d00c7550 2024-03-29T11:29:00.753 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.746+0000 7fa7c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:00.753 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.746+0000 7fa7c3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:00.753 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.746+0000 7fa7c3fff640 10 monclient: dump: 2024-03-29T11:29:00.753 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:29:00.753 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:29:00.753 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:00.753 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:00.754 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:00.754 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:00.754 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:29:00.754 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:29:00.754 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:29:00.754 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:00.754 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.746+0000 7fa7d5ffb640 4 set_mon_vals no callback set 2024-03-29T11:29:00.754 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.748+0000 7fa7ddce6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:00.754 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.748+0000 7fa7ddce6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:00.754 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.748+0000 7fa7ddce6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:29:00.754 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.749+0000 7fa7ddce6640 1 -- 172.21.15.104:0/29993141 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7b8005080 con 0x7fa7d00c7550 2024-03-29T11:29:00.754 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.749+0000 7fa7c3fff640 1 -- 172.21.15.104:0/29993141 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7fa7d8067bf0 con 0x7fa7d00c7550 2024-03-29T11:29:00.754 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.750+0000 7fa7d77fe640 10 monclient: get_auth_request con 0x7fa7bc06de40 auth_method 0 2024-03-29T11:29:00.754 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.750+0000 7fa7c3fff640 10 client.4552.objecter ms_handle_connect 0x7fa7bc06de40 2024-03-29T11:29:00.754 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.750+0000 7fa7c3fff640 1 -- 172.21.15.104:0/29993141 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7fa7d80cdf60 con 0x7fa7d00c7550 2024-03-29T11:29:00.755 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.750+0000 7fa7c3fff640 10 client.4552.objecter ms_dispatch 0x7fa7d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:00.755 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.750+0000 7fa7c3fff640 3 client.4552.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:00.755 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.750+0000 7fa7c3fff640 3 client.4552.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:00.755 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.750+0000 7fa7c3fff640 20 client.4552.objecter dump_active .. 0 homeless 2024-03-29T11:29:00.755 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.757+0000 7fa7c3fff640 1 -- 172.21.15.104:0/29993141 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa7d80ca200 con 0x7fa7d00c7550 2024-03-29T11:29:00.755 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.757+0000 7fa7c3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:00.755 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:00.757+0000 7fa7c3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:01.135 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:01.143+0000 7fa7ddce6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:01.135 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:01.143+0000 7fa7ddce6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:01.135 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:01.143+0000 7fa7ddce6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:29:01.135 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:01.143+0000 7fa7ddce6640 1 -- 172.21.15.104:0/29993141 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa7b8005330 con 0x7fa7d00c7550 2024-03-29T11:29:01.135 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:01.144+0000 7fa7c3fff640 1 -- 172.21.15.104:0/29993141 <== mon.1 v1:172.21.15.119:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7fa7d80ca3a0 con 0x7fa7d00c7550 2024-03-29T11:29:01.135 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:01.144+0000 7fa7c3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:01.135 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:01.144+0000 7fa7c3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:01.136 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:29:01.136 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:01.139 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:01.148+0000 7fa7ddce6640 10 librados: watch_flush enter 2024-03-29T11:29:01.139 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:01.148+0000 7fa7ddce6640 10 librados: watch_flush exit 2024-03-29T11:29:01.139 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:01.148+0000 7fa7ddce6640 20 client.4552.objecter shutdown clearing up homeless session... 2024-03-29T11:29:01.139 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:01.148+0000 7fa7ddce6640 10 client.4552.objecter successfully canceled tick 2024-03-29T11:29:01.139 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:01.148+0000 7fa7ddce6640 1 -- 172.21.15.104:0/29993141 >> v1:172.21.15.104:6816/3341126660 conn(0x7fa7bc06de40 legacy=0x7fa7bc070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:01.139 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:01.148+0000 7fa7ddce6640 10 monclient: shutdown 2024-03-29T11:29:01.139 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:01.148+0000 7fa7ddce6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:01.139 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:01.149+0000 7fa7ddce6640 1 -- 172.21.15.104:0/29993141 >> v1:172.21.15.119:6789/0 conn(0x7fa7d00c7550 legacy=0x7fa7d0163180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:01.140 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:01.149+0000 7fa7ddce6640 1 -- 172.21.15.104:0/29993141 shutdown_connections 2024-03-29T11:29:01.140 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:01.149+0000 7fa7ddce6640 1 -- 172.21.15.104:0/29993141 >> 172.21.15.104:0/29993141 conn(0x7fa7d0005c00 msgr2=0x7fa7d0032620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:01.140 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:01.149+0000 7fa7ddce6640 1 -- 172.21.15.104:0/29993141 shutdown_connections 2024-03-29T11:29:01.140 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:01.149+0000 7fa7ddce6640 1 -- 172.21.15.104:0/29993141 wait complete. 2024-03-29T11:29:01.140 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:01.149+0000 7fa7ddce6640 1 librados: shutdown 2024-03-29T11:29:01.155 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:07.963 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:08.108 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.117+0000 7f6721b15640 10 monclient: get_monmap_and_config 2024-03-29T11:29:08.108 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.117+0000 7f6721b15640 10 monclient: build_initial_monmap 2024-03-29T11:29:08.108 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.117+0000 7f6721b15640 10 monclient: monmap: 2024-03-29T11:29:08.108 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:29:08.108 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:29:08.108 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:29:08.118278+0000 2024-03-29T11:29:08.108 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:29:08.118278+0000 2024-03-29T11:29:08.108 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:08.108 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:08.108 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:29:08.109 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:29:08.109 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:29:08.109 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:08.111 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.119+0000 7f6721b15640 1 Processor -- start 2024-03-29T11:29:08.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.119+0000 7f6721b15640 1 -- start start 2024-03-29T11:29:08.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.119+0000 7f6721b15640 10 monclient: init 2024-03-29T11:29:08.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.120+0000 7f6721b15640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:08.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.120+0000 7f6721b15640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:29:08.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.120+0000 7f6721b15640 10 monclient(hunting): picked mon.noname-b con 0x7f671c0b4920 addr v1:172.21.15.104:6790/0 2024-03-29T11:29:08.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.120+0000 7f6721b15640 10 monclient(hunting): picked mon.noname-c con 0x7f671c0b4200 addr v1:172.21.15.119:6789/0 2024-03-29T11:29:08.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.120+0000 7f6721b15640 10 monclient(hunting): picked mon.noname-a con 0x7f671c13f9a0 addr v1:172.21.15.104:6789/0 2024-03-29T11:29:08.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.120+0000 7f6721b15640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f671c0b0930 con 0x7f671c13f9a0 2024-03-29T11:29:08.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.120+0000 7f6721b15640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f671c0af460 con 0x7f671c0b4200 2024-03-29T11:29:08.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.120+0000 7f6721b15640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f671c0aeda0 con 0x7f671c0b4920 2024-03-29T11:29:08.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.120+0000 7f6721b15640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:08.112 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.120+0000 7f6721b15640 10 monclient(hunting): authenticate will time out at 1529.371769s 2024-03-29T11:29:08.115 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.122+0000 7f671bfff640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f671c0b4200 0x7f671c0b0fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:51904/0 (socket says 172.21.15.104:51904) 2024-03-29T11:29:08.115 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.122+0000 7f671bfff640 1 -- 172.21.15.104:0/3748170335 learned_addr learned my addr 172.21.15.104:0/3748170335 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:29:08.115 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.123+0000 7f671b7fe640 1 -- 172.21.15.104:0/3748170335 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3597168961 0 0) 0x7f671c0af460 con 0x7f671c0b4200 2024-03-29T11:29:08.115 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.123+0000 7f671b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:29:08.115 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.123+0000 7f671b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:08.115 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.123+0000 7f671b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:08.115 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.123+0000 7f671b7fe640 10 monclient(hunting): my global_id is 4555 2024-03-29T11:29:08.116 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.123+0000 7f671b7fe640 1 -- 172.21.15.104:0/3748170335 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6708003520 con 0x7f671c0b4200 2024-03-29T11:29:08.117 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.125+0000 7f671b7fe640 1 -- 172.21.15.104:0/3748170335 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3131897918 0 0) 0x7f6708003520 con 0x7f671c0b4200 2024-03-29T11:29:08.117 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.126+0000 7f671b7fe640 1 -- 172.21.15.104:0/3748170335 >> v1:172.21.15.104:6790/0 conn(0x7f671c0b4920 legacy=0x7f671c13f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:08.117 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.126+0000 7f671b7fe640 1 -- 172.21.15.104:0/3748170335 >> v1:172.21.15.104:6789/0 conn(0x7f671c13f9a0 legacy=0x7f671c14d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:08.117 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.126+0000 7f671b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:29:08.118 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.126+0000 7f671b7fe640 1 monclient: found mon.noname-c 2024-03-29T11:29:08.118 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.126+0000 7f671b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:08.118 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.126+0000 7f671b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.119:6789/0 2024-03-29T11:29:08.118 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.126+0000 7f671b7fe640 1 -- 172.21.15.104:0/3748170335 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f671c0af980 con 0x7f671c0b4200 2024-03-29T11:29:08.118 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.126+0000 7f671b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:29:08.118 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.126+0000 7f671b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:29:08.118 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.126+0000 7f671b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:08.118 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.126+0000 7f671b7fe640 1 -- 172.21.15.104:0/3748170335 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f6704001690 con 0x7f671c0b4200 2024-03-29T11:29:08.118 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.126+0000 7f671b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:08.119 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.126+0000 7f671b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:08.119 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.126+0000 7f671b7fe640 10 monclient: dump: 2024-03-29T11:29:08.119 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:29:08.119 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:29:08.119 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:08.119 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:08.119 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:08.119 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:08.119 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:29:08.120 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:29:08.120 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:29:08.120 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:08.120 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.126+0000 7f6721b15640 5 monclient: authenticate success, global_id 4555 2024-03-29T11:29:08.120 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.126+0000 7f6721b15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:08.121 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.128+0000 7f671b7fe640 1 -- 172.21.15.104:0/3748170335 <== mon.1 v1:172.21.15.119:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6704001860 con 0x7f671c0b4200 2024-03-29T11:29:08.121 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.128+0000 7f671b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:08.122 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.128+0000 7f671b7fe640 1 -- 172.21.15.104:0/3748170335 <== mon.1 v1:172.21.15.119:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f6704001cb0 con 0x7f671c0b4200 2024-03-29T11:29:08.122 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.128+0000 7f671b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:08.122 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.128+0000 7f671b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:08.122 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.128+0000 7f671b7fe640 10 monclient: dump: 2024-03-29T11:29:08.122 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:29:08.122 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:29:08.122 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:08.122 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:08.122 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:08.122 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:08.122 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:29:08.122 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:29:08.122 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:29:08.122 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:08.122 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.128+0000 7f6721b15640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:08.123 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.128+0000 7f6721b15640 4 set_mon_vals no callback set 2024-03-29T11:29:08.123 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.129+0000 7f6721b15640 10 monclient: shutdown 2024-03-29T11:29:08.123 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.129+0000 7f6721b15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:08.123 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.129+0000 7f6721b15640 1 -- 172.21.15.104:0/3748170335 >> v1:172.21.15.119:6789/0 conn(0x7f671c0b4200 legacy=0x7f671c0b0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:08.123 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.130+0000 7f6721b15640 1 -- 172.21.15.104:0/3748170335 shutdown_connections 2024-03-29T11:29:08.123 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.130+0000 7f6721b15640 1 -- 172.21.15.104:0/3748170335 >> 172.21.15.104:0/3748170335 conn(0x7f671c06d230 msgr2=0x7f671c066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:08.123 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.130+0000 7f6721b15640 1 -- 172.21.15.104:0/3748170335 shutdown_connections 2024-03-29T11:29:08.123 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.130+0000 7f6721b15640 1 -- 172.21.15.104:0/3748170335 wait complete. 2024-03-29T11:29:08.123 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.131+0000 7f6721b15640 10 monclient: build_initial_monmap 2024-03-29T11:29:08.124 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.131+0000 7f6721b15640 10 monclient: monmap: 2024-03-29T11:29:08.124 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:29:08.124 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:08.124 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:29:08.124 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:29:08.124 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:08.124 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:08.124 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:29:08.124 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:29:08.124 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:29:08.124 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:08.124 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.132+0000 7f6721b15640 1 librados: starting msgr at 2024-03-29T11:29:08.124 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.132+0000 7f6721b15640 1 librados: starting objecter 2024-03-29T11:29:08.124 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.132+0000 7f6721b15640 1 Processor -- start 2024-03-29T11:29:08.124 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.132+0000 7f6721b15640 1 -- start start 2024-03-29T11:29:08.125 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.132+0000 7f6721b15640 1 librados: setting wanted keys 2024-03-29T11:29:08.125 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.132+0000 7f6721b15640 1 librados: calling monclient init 2024-03-29T11:29:08.125 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.132+0000 7f6721b15640 10 monclient: init 2024-03-29T11:29:08.125 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.133+0000 7f6721b15640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:08.125 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.133+0000 7f6721b15640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:29:08.125 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.133+0000 7f6721b15640 10 monclient(hunting): picked mon.noname-c con 0x7f671c0b4200 addr v1:172.21.15.104:6790/0 2024-03-29T11:29:08.125 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.133+0000 7f6721b15640 10 monclient(hunting): picked mon.noname-b con 0x7f671c0b4920 addr v1:172.21.15.119:6789/0 2024-03-29T11:29:08.125 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.133+0000 7f6721b15640 10 monclient(hunting): picked mon.noname-a con 0x7f671c13f9a0 addr v1:172.21.15.104:6789/0 2024-03-29T11:29:08.125 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.133+0000 7f6721b15640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f671c136f90 con 0x7f671c13f9a0 2024-03-29T11:29:08.126 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.133+0000 7f6721b15640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f671c0a9850 con 0x7f671c0b4920 2024-03-29T11:29:08.126 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.133+0000 7f6721b15640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f671c0a94e0 con 0x7f671c0b4200 2024-03-29T11:29:08.126 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.133+0000 7f6721b15640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:08.126 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.133+0000 7f6721b15640 10 monclient(hunting): authenticate will time out at 1529.384548s 2024-03-29T11:29:08.126 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.134+0000 7f671bfff640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f671c0b4920 0x7f671c136880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:52350/0 (socket says 172.21.15.104:52350) 2024-03-29T11:29:08.126 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.134+0000 7f671bfff640 1 -- 172.21.15.104:0/2808655399 learned_addr learned my addr 172.21.15.104:0/2808655399 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:29:08.130 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.140+0000 7f67197fa640 10 client.?.objecter ms_handle_connect 0x7f671c0b4920 2024-03-29T11:29:08.130 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.140+0000 7f67197fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:08.130 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.140+0000 7f67197fa640 1 -- 172.21.15.104:0/2808655399 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3513124485 0 0) 0x7f671c0a9850 con 0x7f671c0b4920 2024-03-29T11:29:08.130 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.140+0000 7f67197fa640 10 monclient(hunting): _negotiate 2024-03-29T11:29:08.130 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.140+0000 7f67197fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:08.130 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.140+0000 7f67197fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:08.131 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.140+0000 7f67197fa640 10 monclient(hunting): my global_id is 4558 2024-03-29T11:29:08.131 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.140+0000 7f67197fa640 1 -- 172.21.15.104:0/2808655399 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66fc004eb0 con 0x7f671c0b4920 2024-03-29T11:29:08.132 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.140+0000 7f67197fa640 10 client.?.objecter ms_handle_connect 0x7f671c0b4200 2024-03-29T11:29:08.132 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.140+0000 7f67197fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:08.132 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.140+0000 7f67197fa640 10 client.?.objecter ms_handle_connect 0x7f671c13f9a0 2024-03-29T11:29:08.132 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.140+0000 7f67197fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:08.132 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.140+0000 7f67197fa640 1 -- 172.21.15.104:0/2808655399 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2995985013 0 0) 0x7f66fc004eb0 con 0x7f671c0b4920 2024-03-29T11:29:08.132 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.140+0000 7f67197fa640 1 -- 172.21.15.104:0/2808655399 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f671c0a9850 con 0x7f671c0b4920 2024-03-29T11:29:08.133 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.140+0000 7f67197fa640 1 -- 172.21.15.104:0/2808655399 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f6704001b70 con 0x7f671c0b4920 2024-03-29T11:29:08.133 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.140+0000 7f67197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:08.133 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.140+0000 7f67197fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:08.133 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.140+0000 7f67197fa640 10 monclient(hunting): dump: 2024-03-29T11:29:08.133 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:29:08.133 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:29:08.133 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:08.133 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:08.133 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:08.133 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:08.133 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:29:08.133 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:29:08.133 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:29:08.133 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:08.134 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.140+0000 7f67197fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:08.134 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.140+0000 7f67197fa640 1 -- 172.21.15.104:0/2808655399 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1140514618 0 0) 0x7f671c0a94e0 con 0x7f671c0b4200 2024-03-29T11:29:08.134 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.140+0000 7f67197fa640 10 monclient(hunting): _negotiate 2024-03-29T11:29:08.134 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.140+0000 7f67197fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:08.134 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.140+0000 7f67197fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:08.134 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.140+0000 7f67197fa640 10 monclient(hunting): my global_id is 4607 2024-03-29T11:29:08.134 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.140+0000 7f67197fa640 1 -- 172.21.15.104:0/2808655399 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66fc004eb0 con 0x7f671c0b4200 2024-03-29T11:29:08.134 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.140+0000 7f67197fa640 1 -- 172.21.15.104:0/2808655399 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2466538629 0 0) 0x7f6714000fa0 con 0x7f671c13f9a0 2024-03-29T11:29:08.134 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.140+0000 7f67197fa640 10 monclient(hunting): _negotiate 2024-03-29T11:29:08.134 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.140+0000 7f67197fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:08.134 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.140+0000 7f67197fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:08.134 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.140+0000 7f67197fa640 10 monclient(hunting): my global_id is 4566 2024-03-29T11:29:08.134 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.140+0000 7f67197fa640 1 -- 172.21.15.104:0/2808655399 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f671c0a94e0 con 0x7f671c13f9a0 2024-03-29T11:29:08.134 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.140+0000 7f67197fa640 1 -- 172.21.15.104:0/2808655399 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3670343127 0 0) 0x7f671c0a9850 con 0x7f671c0b4920 2024-03-29T11:29:08.135 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.140+0000 7f67197fa640 1 -- 172.21.15.104:0/2808655399 >> v1:172.21.15.104:6790/0 conn(0x7f671c0b4200 legacy=0x7f671c136020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:08.135 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.140+0000 7f67197fa640 1 -- 172.21.15.104:0/2808655399 >> v1:172.21.15.104:6789/0 conn(0x7f671c13f9a0 legacy=0x7f671c1ffaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:08.135 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.140+0000 7f67197fa640 10 monclient: _finish_hunting 0 2024-03-29T11:29:08.135 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.140+0000 7f67197fa640 1 monclient: found mon.b 2024-03-29T11:29:08.135 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.140+0000 7f67197fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:08.135 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.140+0000 7f67197fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:29:08.135 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.140+0000 7f67197fa640 1 -- 172.21.15.104:0/2808655399 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f671c0aa800 con 0x7f671c0b4920 2024-03-29T11:29:08.135 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.140+0000 7f67197fa640 10 monclient: _finish_auth 0 2024-03-29T11:29:08.135 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.140+0000 7f67197fa640 10 monclient: _check_auth_tickets 2024-03-29T11:29:08.135 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.140+0000 7f67197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:08.135 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.140+0000 7f6721b15640 5 monclient: authenticate success, global_id 4558 2024-03-29T11:29:08.135 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.140+0000 7f6721b15640 10 monclient: _renew_subs 2024-03-29T11:29:08.135 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.140+0000 7f6721b15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:29:08.136 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.140+0000 7f6721b15640 1 -- 172.21.15.104:0/2808655399 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f671c0a9cc0 con 0x7f671c0b4920 2024-03-29T11:29:08.136 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.141+0000 7f6721b15640 10 client.4558.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:08.136 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.141+0000 7f6721b15640 10 monclient: _renew_subs 2024-03-29T11:29:08.136 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.141+0000 7f6721b15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:29:08.136 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.141+0000 7f6721b15640 1 -- 172.21.15.104:0/2808655399 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f671c06f850 con 0x7f671c0b4920 2024-03-29T11:29:08.136 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.141+0000 7f6721b15640 1 librados: init done 2024-03-29T11:29:08.136 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.141+0000 7f67197fa640 1 -- 172.21.15.104:0/2808655399 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6704004bf0 con 0x7f671c0b4920 2024-03-29T11:29:08.136 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.141+0000 7f67197fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:08.136 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.141+0000 7f67197fa640 1 -- 172.21.15.104:0/2808655399 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f6704004fc0 con 0x7f671c0b4920 2024-03-29T11:29:08.137 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.141+0000 7f67197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:08.137 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.141+0000 7f67197fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:08.137 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.141+0000 7f671affd640 4 set_mon_vals no callback set 2024-03-29T11:29:08.137 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.141+0000 7f67197fa640 10 monclient: dump: 2024-03-29T11:29:08.137 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:29:08.137 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:29:08.137 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:08.137 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:08.137 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:08.137 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:08.137 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:29:08.137 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:29:08.137 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:29:08.137 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:08.138 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.142+0000 7f6721b15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:08.138 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.142+0000 7f6721b15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:08.138 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.142+0000 7f6721b15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:29:08.138 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.142+0000 7f6721b15640 1 -- 172.21.15.104:0/2808655399 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6700005080 con 0x7f671c0b4920 2024-03-29T11:29:08.138 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.143+0000 7f67197fa640 1 -- 172.21.15.104:0/2808655399 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f6704005120 con 0x7f671c0b4920 2024-03-29T11:29:08.138 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.144+0000 7f67197fa640 1 -- 172.21.15.104:0/2808655399 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7f6704084d00 con 0x7f671c0b4920 2024-03-29T11:29:08.138 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.144+0000 7f67197fa640 10 client.4558.objecter ms_dispatch 0x7f671c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:08.138 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.144+0000 7f67197fa640 3 client.4558.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:08.138 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.144+0000 7f67197fa640 3 client.4558.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:08.138 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.144+0000 7f67197fa640 20 client.4558.objecter dump_active .. 0 homeless 2024-03-29T11:29:08.138 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.144+0000 7f6720b13640 10 monclient: get_auth_request con 0x7f66fc06dea0 auth_method 0 2024-03-29T11:29:08.139 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.144+0000 7f67197fa640 10 client.4558.objecter ms_handle_connect 0x7f66fc06dea0 2024-03-29T11:29:08.141 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.151+0000 7f67197fa640 1 -- 172.21.15.104:0/2808655399 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f67040505e0 con 0x7f671c0b4920 2024-03-29T11:29:08.141 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.151+0000 7f67197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:08.141 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.151+0000 7f67197fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:08.463 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.472+0000 7f6721b15640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:08.463 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.472+0000 7f6721b15640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:08.463 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.472+0000 7f6721b15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:29:08.464 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.472+0000 7f6721b15640 1 -- 172.21.15.104:0/2808655399 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6700005330 con 0x7f671c0b4920 2024-03-29T11:29:08.465 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.474+0000 7f67197fa640 1 -- 172.21.15.104:0/2808655399 <== mon.1 v1:172.21.15.119:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7f6704052510 con 0x7f671c0b4920 2024-03-29T11:29:08.465 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.474+0000 7f67197fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:08.465 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.474+0000 7f67197fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:08.465 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:29:08.465 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:08.469 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.478+0000 7f6721b15640 10 librados: watch_flush enter 2024-03-29T11:29:08.469 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.478+0000 7f6721b15640 10 librados: watch_flush exit 2024-03-29T11:29:08.469 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.478+0000 7f6721b15640 20 client.4558.objecter shutdown clearing up homeless session... 2024-03-29T11:29:08.469 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.478+0000 7f6721b15640 10 client.4558.objecter successfully canceled tick 2024-03-29T11:29:08.470 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.478+0000 7f6721b15640 1 -- 172.21.15.104:0/2808655399 >> v1:172.21.15.104:6816/3341126660 conn(0x7f66fc06dea0 legacy=0x7f66fc070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:08.470 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.478+0000 7f6721b15640 10 monclient: shutdown 2024-03-29T11:29:08.470 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.478+0000 7f6721b15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:08.470 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.478+0000 7f6721b15640 1 -- 172.21.15.104:0/2808655399 >> v1:172.21.15.119:6789/0 conn(0x7f671c0b4920 legacy=0x7f671c136880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:08.470 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.478+0000 7f6721b15640 1 -- 172.21.15.104:0/2808655399 shutdown_connections 2024-03-29T11:29:08.470 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.478+0000 7f6721b15640 1 -- 172.21.15.104:0/2808655399 >> 172.21.15.104:0/2808655399 conn(0x7f671c06d230 msgr2=0x7f671c0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:08.470 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.479+0000 7f6721b15640 1 -- 172.21.15.104:0/2808655399 shutdown_connections 2024-03-29T11:29:08.470 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.479+0000 7f6721b15640 1 -- 172.21.15.104:0/2808655399 wait complete. 2024-03-29T11:29:08.470 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:08.479+0000 7f6721b15640 1 librados: shutdown 2024-03-29T11:29:08.487 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:15.295 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:15.425 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.434+0000 7fb682510640 10 monclient: get_monmap_and_config 2024-03-29T11:29:15.426 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.434+0000 7fb682510640 10 monclient: build_initial_monmap 2024-03-29T11:29:15.426 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.434+0000 7fb682510640 10 monclient: monmap: 2024-03-29T11:29:15.426 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:29:15.426 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:29:15.426 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:29:15.435763+0000 2024-03-29T11:29:15.426 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:29:15.435763+0000 2024-03-29T11:29:15.426 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:15.426 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:15.426 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:29:15.426 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:29:15.426 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:29:15.426 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:15.427 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.436+0000 7fb682510640 1 Processor -- start 2024-03-29T11:29:15.428 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.436+0000 7fb682510640 1 -- start start 2024-03-29T11:29:15.428 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.436+0000 7fb682510640 10 monclient: init 2024-03-29T11:29:15.428 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.437+0000 7fb682510640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:15.428 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.437+0000 7fb682510640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:29:15.428 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.437+0000 7fb682510640 10 monclient(hunting): picked mon.noname-a con 0x7fb67c1457f0 addr v1:172.21.15.104:6789/0 2024-03-29T11:29:15.428 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.437+0000 7fb682510640 10 monclient(hunting): picked mon.noname-c con 0x7fb67c13f650 addr v1:172.21.15.119:6789/0 2024-03-29T11:29:15.428 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.437+0000 7fb682510640 10 monclient(hunting): picked mon.noname-b con 0x7fb67c0a9d70 addr v1:172.21.15.104:6790/0 2024-03-29T11:29:15.428 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.437+0000 7fb682510640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb67c0aaef0 con 0x7fb67c1457f0 2024-03-29T11:29:15.428 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.437+0000 7fb682510640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb67c145bd0 con 0x7fb67c13f650 2024-03-29T11:29:15.428 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.437+0000 7fb682510640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb67c145eb0 con 0x7fb67c0a9d70 2024-03-29T11:29:15.428 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.437+0000 7fb682510640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:15.428 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.437+0000 7fb682510640 10 monclient(hunting): authenticate will time out at 1536.688227s 2024-03-29T11:29:15.428 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.437+0000 7fb680aad640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7fb67c0a9d70 0x7fb67c14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:47784/0 (socket says 172.21.15.104:47784) 2024-03-29T11:29:15.429 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.437+0000 7fb680aad640 1 -- 172.21.15.104:0/2812906363 learned_addr learned my addr 172.21.15.104:0/2812906363 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:29:15.432 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.437+0000 7fb67affd640 1 -- 172.21.15.104:0/2812906363 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1694166276 0 0) 0x7fb67c145eb0 con 0x7fb67c0a9d70 2024-03-29T11:29:15.432 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.437+0000 7fb67affd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:15.432 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.437+0000 7fb67affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:15.432 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.437+0000 7fb67affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:15.432 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.437+0000 7fb67affd640 10 monclient(hunting): my global_id is 4610 2024-03-29T11:29:15.432 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.437+0000 7fb67affd640 1 -- 172.21.15.104:0/2812906363 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb664003520 con 0x7fb67c0a9d70 2024-03-29T11:29:15.432 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.437+0000 7fb67affd640 1 -- 172.21.15.104:0/2812906363 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 255909786 0 0) 0x7fb67c145bd0 con 0x7fb67c13f650 2024-03-29T11:29:15.432 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.437+0000 7fb67affd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:15.432 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.437+0000 7fb67affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:15.432 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.437+0000 7fb67affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:15.432 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.437+0000 7fb67affd640 10 monclient(hunting): my global_id is 4561 2024-03-29T11:29:15.432 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.437+0000 7fb67affd640 1 -- 172.21.15.104:0/2812906363 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb67c145eb0 con 0x7fb67c13f650 2024-03-29T11:29:15.432 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.438+0000 7fb67affd640 1 -- 172.21.15.104:0/2812906363 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 857808576 0 0) 0x7fb664003520 con 0x7fb67c0a9d70 2024-03-29T11:29:15.433 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.438+0000 7fb67affd640 1 -- 172.21.15.104:0/2812906363 >> v1:172.21.15.119:6789/0 conn(0x7fb67c13f650 legacy=0x7fb67c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:15.433 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.438+0000 7fb67affd640 1 -- 172.21.15.104:0/2812906363 >> v1:172.21.15.104:6789/0 conn(0x7fb67c1457f0 legacy=0x7fb67c13ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:15.433 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.439+0000 7fb67affd640 10 monclient: _finish_hunting 0 2024-03-29T11:29:15.433 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.439+0000 7fb67affd640 1 monclient: found mon.noname-b 2024-03-29T11:29:15.433 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.439+0000 7fb67affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:15.433 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.439+0000 7fb67affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.104:6790/0 2024-03-29T11:29:15.433 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.439+0000 7fb67affd640 1 -- 172.21.15.104:0/2812906363 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb67c146340 con 0x7fb67c0a9d70 2024-03-29T11:29:15.433 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.439+0000 7fb67affd640 10 monclient: _finish_auth 0 2024-03-29T11:29:15.433 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.439+0000 7fb67affd640 10 monclient: _check_auth_tickets 2024-03-29T11:29:15.433 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.439+0000 7fb67affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:15.433 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.439+0000 7fb682510640 5 monclient: authenticate success, global_id 4610 2024-03-29T11:29:15.433 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.439+0000 7fb682510640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:15.433 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.439+0000 7fb67affd640 1 -- 172.21.15.104:0/2812906363 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fb66c001690 con 0x7fb67c0a9d70 2024-03-29T11:29:15.433 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.439+0000 7fb67affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:15.433 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.439+0000 7fb67affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:15.434 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.439+0000 7fb67affd640 10 monclient: dump: 2024-03-29T11:29:15.434 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:29:15.434 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:29:15.435 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:15.435 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:15.435 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:15.435 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:15.435 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:29:15.435 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:29:15.436 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:29:15.436 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:15.436 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.439+0000 7fb682510640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:15.436 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.440+0000 7fb67affd640 1 -- 172.21.15.104:0/2812906363 <== mon.2 v1:172.21.15.104:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb66c0017f0 con 0x7fb67c0a9d70 2024-03-29T11:29:15.436 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.440+0000 7fb67affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:15.436 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.440+0000 7fb67affd640 1 -- 172.21.15.104:0/2812906363 <== mon.2 v1:172.21.15.104:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fb66c001c40 con 0x7fb67c0a9d70 2024-03-29T11:29:15.437 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.440+0000 7fb682510640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:15.437 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.440+0000 7fb682510640 4 set_mon_vals no callback set 2024-03-29T11:29:15.437 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.440+0000 7fb67affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:15.437 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.440+0000 7fb67affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:15.437 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.440+0000 7fb67affd640 10 monclient: dump: 2024-03-29T11:29:15.437 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:29:15.437 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:29:15.437 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:15.438 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:15.438 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:15.438 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:15.438 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:29:15.438 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:29:15.438 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:29:15.438 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:15.438 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.441+0000 7fb682510640 10 monclient: shutdown 2024-03-29T11:29:15.438 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.441+0000 7fb682510640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:15.438 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.441+0000 7fb682510640 1 -- 172.21.15.104:0/2812906363 >> v1:172.21.15.104:6790/0 conn(0x7fb67c0a9d70 legacy=0x7fb67c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:15.438 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.441+0000 7fb682510640 1 -- 172.21.15.104:0/2812906363 shutdown_connections 2024-03-29T11:29:15.438 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.441+0000 7fb682510640 1 -- 172.21.15.104:0/2812906363 >> 172.21.15.104:0/2812906363 conn(0x7fb67c066a00 msgr2=0x7fb67c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:15.438 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.441+0000 7fb682510640 1 -- 172.21.15.104:0/2812906363 shutdown_connections 2024-03-29T11:29:15.439 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.441+0000 7fb682510640 1 -- 172.21.15.104:0/2812906363 wait complete. 2024-03-29T11:29:15.450 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.442+0000 7fb682510640 10 monclient: build_initial_monmap 2024-03-29T11:29:15.450 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.442+0000 7fb682510640 10 monclient: monmap: 2024-03-29T11:29:15.450 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:29:15.450 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:15.450 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:29:15.451 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:29:15.451 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:15.451 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:15.451 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:29:15.451 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:29:15.451 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:29:15.451 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:15.452 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.442+0000 7fb682510640 1 librados: starting msgr at 2024-03-29T11:29:15.452 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.442+0000 7fb682510640 1 librados: starting objecter 2024-03-29T11:29:15.452 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.442+0000 7fb682510640 1 Processor -- start 2024-03-29T11:29:15.452 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.442+0000 7fb682510640 1 -- start start 2024-03-29T11:29:15.452 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.442+0000 7fb682510640 1 librados: setting wanted keys 2024-03-29T11:29:15.452 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.442+0000 7fb682510640 1 librados: calling monclient init 2024-03-29T11:29:15.452 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.442+0000 7fb682510640 10 monclient: init 2024-03-29T11:29:15.452 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.443+0000 7fb682510640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:15.452 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.443+0000 7fb682510640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:29:15.453 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.443+0000 7fb682510640 10 monclient(hunting): picked mon.noname-a con 0x7fb67c0a9d70 addr v1:172.21.15.104:6789/0 2024-03-29T11:29:15.453 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.443+0000 7fb682510640 10 monclient(hunting): picked mon.noname-c con 0x7fb67c13f650 addr v1:172.21.15.104:6790/0 2024-03-29T11:29:15.453 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.443+0000 7fb682510640 10 monclient(hunting): picked mon.noname-b con 0x7fb67c1457f0 addr v1:172.21.15.119:6789/0 2024-03-29T11:29:15.453 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.443+0000 7fb682510640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb67c1418c0 con 0x7fb67c0a9d70 2024-03-29T11:29:15.453 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.443+0000 7fb682510640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb67c1d7010 con 0x7fb67c1457f0 2024-03-29T11:29:15.453 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.443+0000 7fb682510640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb67c13b840 con 0x7fb67c13f650 2024-03-29T11:29:15.453 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.443+0000 7fb682510640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:15.453 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.443+0000 7fb682510640 10 monclient(hunting): authenticate will time out at 1536.694845s 2024-03-29T11:29:15.453 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.444+0000 7fb67b7fe640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7fb67c13f650 0x7fb67c1423d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:48734/0 (socket says 172.21.15.104:48734) 2024-03-29T11:29:15.453 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.444+0000 7fb67b7fe640 1 -- 172.21.15.104:0/923067586 learned_addr learned my addr 172.21.15.104:0/923067586 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:29:15.453 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.444+0000 7fb678ff9640 10 client.?.objecter ms_handle_connect 0x7fb67c13f650 2024-03-29T11:29:15.453 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.444+0000 7fb678ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:15.453 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.444+0000 7fb678ff9640 1 -- 172.21.15.104:0/923067586 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2693349456 0 0) 0x7fb67c13b840 con 0x7fb67c13f650 2024-03-29T11:29:15.453 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.444+0000 7fb678ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:29:15.454 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.444+0000 7fb678ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:15.454 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.444+0000 7fb678ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:15.454 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.444+0000 7fb678ff9640 10 monclient(hunting): my global_id is 4613 2024-03-29T11:29:15.454 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.444+0000 7fb678ff9640 1 -- 172.21.15.104:0/923067586 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb668004eb0 con 0x7fb67c13f650 2024-03-29T11:29:15.454 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.444+0000 7fb678ff9640 1 -- 172.21.15.104:0/923067586 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 979343709 0 0) 0x7fb668004eb0 con 0x7fb67c13f650 2024-03-29T11:29:15.454 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.444+0000 7fb678ff9640 1 -- 172.21.15.104:0/923067586 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb67c13b840 con 0x7fb67c13f650 2024-03-29T11:29:15.454 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.444+0000 7fb678ff9640 1 -- 172.21.15.104:0/923067586 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fb674001dd0 con 0x7fb67c13f650 2024-03-29T11:29:15.454 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.444+0000 7fb678ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:15.454 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.444+0000 7fb678ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:15.455 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.444+0000 7fb678ff9640 10 monclient(hunting): dump: 2024-03-29T11:29:15.455 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:29:15.455 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:29:15.455 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:15.455 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:15.455 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:15.455 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:15.455 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:29:15.455 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:29:15.456 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:29:15.456 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:15.456 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.444+0000 7fb678ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:15.456 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.444+0000 7fb678ff9640 1 -- 172.21.15.104:0/923067586 <== mon.2 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1439484155 0 0) 0x7fb67c13b840 con 0x7fb67c13f650 2024-03-29T11:29:15.456 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.444+0000 7fb678ff9640 1 -- 172.21.15.104:0/923067586 >> v1:172.21.15.119:6789/0 conn(0x7fb67c1457f0 legacy=0x7fb67c1e2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:15.456 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.444+0000 7fb678ff9640 1 -- 172.21.15.104:0/923067586 >> v1:172.21.15.104:6789/0 conn(0x7fb67c0a9d70 legacy=0x7fb67c1d7530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:29:15.456 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.446+0000 7fb678ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:29:15.456 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.446+0000 7fb678ff9640 1 monclient: found mon.c 2024-03-29T11:29:15.456 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.446+0000 7fb678ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:15.456 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.446+0000 7fb678ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:29:15.456 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.446+0000 7fb678ff9640 1 -- 172.21.15.104:0/923067586 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb67c141ce0 con 0x7fb67c13f650 2024-03-29T11:29:15.456 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.446+0000 7fb678ff9640 10 monclient: _finish_auth 0 2024-03-29T11:29:15.456 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.446+0000 7fb678ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:29:15.457 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.446+0000 7fb678ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:15.457 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.446+0000 7fb682510640 5 monclient: authenticate success, global_id 4613 2024-03-29T11:29:15.457 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.447+0000 7fb682510640 10 monclient: _renew_subs 2024-03-29T11:29:15.457 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.447+0000 7fb682510640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:29:15.457 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.447+0000 7fb682510640 1 -- 172.21.15.104:0/923067586 --> v1:172.21.15.104:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb67c1d7c40 con 0x7fb67c13f650 2024-03-29T11:29:15.457 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.447+0000 7fb682510640 10 client.4613.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:15.457 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.447+0000 7fb682510640 10 monclient: _renew_subs 2024-03-29T11:29:15.457 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.447+0000 7fb682510640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:29:15.457 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.447+0000 7fb682510640 1 -- 172.21.15.104:0/923067586 --> v1:172.21.15.104:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb67c142cc0 con 0x7fb67c13f650 2024-03-29T11:29:15.457 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.447+0000 7fb682510640 1 librados: init done 2024-03-29T11:29:15.457 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.448+0000 7fb678ff9640 1 -- 172.21.15.104:0/923067586 <== mon.2 v1:172.21.15.104:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb674002030 con 0x7fb67c13f650 2024-03-29T11:29:15.457 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.448+0000 7fb678ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:15.457 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.448+0000 7fb678ff9640 1 -- 172.21.15.104:0/923067586 <== mon.2 v1:172.21.15.104:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fb674003dc0 con 0x7fb67c13f650 2024-03-29T11:29:15.458 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.448+0000 7fb678ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:15.458 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.448+0000 7fb678ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:15.458 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.448+0000 7fb67a7fc640 4 set_mon_vals no callback set 2024-03-29T11:29:15.458 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.448+0000 7fb678ff9640 10 monclient: dump: 2024-03-29T11:29:15.458 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:29:15.458 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:29:15.458 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:15.458 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:15.458 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:15.458 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:15.458 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:29:15.458 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:29:15.458 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:29:15.458 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:15.459 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.448+0000 7fb682510640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:15.459 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.449+0000 7fb682510640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:15.459 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.449+0000 7fb682510640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:29:15.459 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.449+0000 7fb682510640 1 -- 172.21.15.104:0/923067586 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb65c005080 con 0x7fb67c13f650 2024-03-29T11:29:15.459 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.450+0000 7fb678ff9640 1 -- 172.21.15.104:0/923067586 <== mon.2 v1:172.21.15.104:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7fb674003f20 con 0x7fb67c13f650 2024-03-29T11:29:15.459 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.450+0000 7fb678ff9640 1 -- 172.21.15.104:0/923067586 <== mon.2 v1:172.21.15.104:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7fb674084ed0 con 0x7fb67c13f650 2024-03-29T11:29:15.459 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.450+0000 7fb678ff9640 10 client.4613.objecter ms_dispatch 0x7fb67c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:15.459 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.450+0000 7fb678ff9640 3 client.4613.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:15.459 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.450+0000 7fb678ff9640 3 client.4613.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:15.459 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.450+0000 7fb678ff9640 20 client.4613.objecter dump_active .. 0 homeless 2024-03-29T11:29:15.459 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.450+0000 7fb67bfff640 10 monclient: get_auth_request con 0x7fb66806dba0 auth_method 0 2024-03-29T11:29:15.459 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.451+0000 7fb678ff9640 10 client.4613.objecter ms_handle_connect 0x7fb66806dba0 2024-03-29T11:29:15.459 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.457+0000 7fb678ff9640 1 -- 172.21.15.104:0/923067586 <== mon.2 v1:172.21.15.104:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb6740507b0 con 0x7fb67c13f650 2024-03-29T11:29:15.459 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.457+0000 7fb678ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:15.459 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.457+0000 7fb678ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:15.752 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.761+0000 7fb682510640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:15.752 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.761+0000 7fb682510640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:15.752 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.761+0000 7fb682510640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:29:15.753 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.761+0000 7fb682510640 1 -- 172.21.15.104:0/923067586 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb65c005330 con 0x7fb67c13f650 2024-03-29T11:29:15.753 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.762+0000 7fb678ff9640 1 -- 172.21.15.104:0/923067586 <== mon.2 v1:172.21.15.104:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7fb6740507b0 con 0x7fb67c13f650 2024-03-29T11:29:15.753 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.762+0000 7fb678ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:15.753 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.762+0000 7fb678ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:15.753 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:29:15.754 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:15.756 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.765+0000 7fb682510640 10 librados: watch_flush enter 2024-03-29T11:29:15.756 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.765+0000 7fb682510640 10 librados: watch_flush exit 2024-03-29T11:29:15.756 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.765+0000 7fb682510640 20 client.4613.objecter shutdown clearing up homeless session... 2024-03-29T11:29:15.756 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.765+0000 7fb682510640 10 client.4613.objecter successfully canceled tick 2024-03-29T11:29:15.756 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.765+0000 7fb682510640 1 -- 172.21.15.104:0/923067586 >> v1:172.21.15.104:6816/3341126660 conn(0x7fb66806dba0 legacy=0x7fb668070060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:15.756 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.765+0000 7fb682510640 10 monclient: shutdown 2024-03-29T11:29:15.756 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.765+0000 7fb682510640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:15.756 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.765+0000 7fb682510640 1 -- 172.21.15.104:0/923067586 >> v1:172.21.15.104:6790/0 conn(0x7fb67c13f650 legacy=0x7fb67c1423d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:15.757 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.766+0000 7fb682510640 1 -- 172.21.15.104:0/923067586 shutdown_connections 2024-03-29T11:29:15.757 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.766+0000 7fb682510640 1 -- 172.21.15.104:0/923067586 >> 172.21.15.104:0/923067586 conn(0x7fb67c066a00 msgr2=0x7fb67c0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:15.757 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.766+0000 7fb682510640 1 -- 172.21.15.104:0/923067586 shutdown_connections 2024-03-29T11:29:15.757 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.766+0000 7fb682510640 1 -- 172.21.15.104:0/923067586 wait complete. 2024-03-29T11:29:15.757 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:15.767+0000 7fb682510640 1 librados: shutdown 2024-03-29T11:29:15.773 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:22.580 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:22.710 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.719+0000 7feeeccd7640 10 monclient: get_monmap_and_config 2024-03-29T11:29:22.711 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.719+0000 7feeeccd7640 10 monclient: build_initial_monmap 2024-03-29T11:29:22.711 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.719+0000 7feeeccd7640 10 monclient: monmap: 2024-03-29T11:29:22.711 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:29:22.711 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:29:22.711 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:29:22.720809+0000 2024-03-29T11:29:22.711 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:29:22.720809+0000 2024-03-29T11:29:22.711 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:22.711 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:22.711 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:29:22.713 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:29:22.713 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:29:22.713 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:22.728 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.720+0000 7feeeccd7640 1 Processor -- start 2024-03-29T11:29:22.728 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.721+0000 7feeeccd7640 1 -- start start 2024-03-29T11:29:22.728 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.721+0000 7feeeccd7640 10 monclient: init 2024-03-29T11:29:22.728 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.722+0000 7feeeccd7640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:22.728 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.722+0000 7feeeccd7640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:29:22.728 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.722+0000 7feeeccd7640 10 monclient(hunting): picked mon.noname-b con 0x7feee00389e0 addr v1:172.21.15.104:6790/0 2024-03-29T11:29:22.728 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.722+0000 7feeeccd7640 10 monclient(hunting): picked mon.noname-a con 0x7feee0038dc0 addr v1:172.21.15.104:6789/0 2024-03-29T11:29:22.728 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.722+0000 7feeeccd7640 10 monclient(hunting): picked mon.noname-c con 0x7feee0030660 addr v1:172.21.15.119:6789/0 2024-03-29T11:29:22.728 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.722+0000 7feeeccd7640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feee00c54c0 con 0x7feee0038dc0 2024-03-29T11:29:22.728 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.722+0000 7feeeccd7640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feee0036430 con 0x7feee0030660 2024-03-29T11:29:22.729 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.722+0000 7feeeccd7640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feee0039840 con 0x7feee00389e0 2024-03-29T11:29:22.729 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.722+0000 7feeeccd7640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:22.729 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.722+0000 7feeeccd7640 10 monclient(hunting): authenticate will time out at 1543.973418s 2024-03-29T11:29:22.729 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.722+0000 7feee7fff640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7feee0030660 0x7feee00d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:55980/0 (socket says 172.21.15.104:55980) 2024-03-29T11:29:22.729 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.722+0000 7feee7fff640 1 -- 172.21.15.104:0/2428176566 learned_addr learned my addr 172.21.15.104:0/2428176566 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:29:22.729 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.723+0000 7feee67fc640 1 -- 172.21.15.104:0/2428176566 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3113662379 0 0) 0x7feee0039840 con 0x7feee00389e0 2024-03-29T11:29:22.729 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.723+0000 7feee67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:29:22.729 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.723+0000 7feee67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:22.729 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.723+0000 7feee67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:22.729 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.723+0000 7feee67fc640 10 monclient(hunting): my global_id is 4616 2024-03-29T11:29:22.729 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.723+0000 7feee67fc640 1 -- 172.21.15.104:0/2428176566 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feed8003520 con 0x7feee00389e0 2024-03-29T11:29:22.729 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.723+0000 7feee67fc640 1 -- 172.21.15.104:0/2428176566 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1064160219 0 0) 0x7feee0036430 con 0x7feee0030660 2024-03-29T11:29:22.729 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.723+0000 7feee67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:29:22.729 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.723+0000 7feee67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:22.730 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.723+0000 7feee67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:22.730 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.723+0000 7feee67fc640 10 monclient(hunting): my global_id is 4564 2024-03-29T11:29:22.730 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.723+0000 7feee67fc640 1 -- 172.21.15.104:0/2428176566 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feee0039840 con 0x7feee0030660 2024-03-29T11:29:22.730 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.723+0000 7feee67fc640 1 -- 172.21.15.104:0/2428176566 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1179425629 0 0) 0x7feed8003520 con 0x7feee00389e0 2024-03-29T11:29:22.730 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.723+0000 7feee67fc640 1 -- 172.21.15.104:0/2428176566 >> v1:172.21.15.119:6789/0 conn(0x7feee0030660 legacy=0x7feee00d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:22.730 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.723+0000 7feee67fc640 1 -- 172.21.15.104:0/2428176566 >> v1:172.21.15.104:6789/0 conn(0x7feee0038dc0 legacy=0x7feee00cadd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:22.730 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.724+0000 7feee67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:29:22.730 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.724+0000 7feee67fc640 1 monclient: found mon.noname-b 2024-03-29T11:29:22.730 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.724+0000 7feee67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:22.730 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.724+0000 7feee67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.104:6790/0 2024-03-29T11:29:22.730 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.724+0000 7feee67fc640 1 -- 172.21.15.104:0/2428176566 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee0035690 con 0x7feee00389e0 2024-03-29T11:29:22.730 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.724+0000 7feee67fc640 10 monclient: _finish_auth 0 2024-03-29T11:29:22.730 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.724+0000 7feee67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:29:22.730 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.724+0000 7feee67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:22.731 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.724+0000 7feee67fc640 1 -- 172.21.15.104:0/2428176566 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7feedc001690 con 0x7feee00389e0 2024-03-29T11:29:22.731 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.724+0000 7feee67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:22.731 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.724+0000 7feee67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:22.731 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.724+0000 7feee67fc640 10 monclient: dump: 2024-03-29T11:29:22.731 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:29:22.731 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:29:22.731 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:22.731 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:22.732 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:22.732 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:22.732 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:29:22.732 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:29:22.732 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:29:22.732 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:22.732 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.725+0000 7feeeccd7640 5 monclient: authenticate success, global_id 4616 2024-03-29T11:29:22.732 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.725+0000 7feeeccd7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:22.732 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.725+0000 7feee67fc640 1 -- 172.21.15.104:0/2428176566 <== mon.2 v1:172.21.15.104:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feedc0018f0 con 0x7feee00389e0 2024-03-29T11:29:22.732 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.725+0000 7feee67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:22.732 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.725+0000 7feee67fc640 1 -- 172.21.15.104:0/2428176566 <== mon.2 v1:172.21.15.104:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7feedc001d40 con 0x7feee00389e0 2024-03-29T11:29:22.732 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.725+0000 7feee67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:22.732 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.725+0000 7feee67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:22.732 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.725+0000 7feee67fc640 10 monclient: dump: 2024-03-29T11:29:22.733 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:29:22.733 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:29:22.733 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:22.733 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:22.733 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:22.733 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:22.733 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:29:22.733 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:29:22.733 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:29:22.733 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:22.733 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.725+0000 7feeeccd7640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:22.733 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.725+0000 7feeeccd7640 4 set_mon_vals no callback set 2024-03-29T11:29:22.733 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.726+0000 7feeeccd7640 10 monclient: shutdown 2024-03-29T11:29:22.733 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.726+0000 7feeeccd7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:22.733 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.726+0000 7feeeccd7640 1 -- 172.21.15.104:0/2428176566 >> v1:172.21.15.104:6790/0 conn(0x7feee00389e0 legacy=0x7feee0033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:22.734 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.726+0000 7feeeccd7640 1 -- 172.21.15.104:0/2428176566 shutdown_connections 2024-03-29T11:29:22.734 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.726+0000 7feeeccd7640 1 -- 172.21.15.104:0/2428176566 >> 172.21.15.104:0/2428176566 conn(0x7feee0005c00 msgr2=0x7feee0006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:22.734 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.727+0000 7feeeccd7640 1 -- 172.21.15.104:0/2428176566 shutdown_connections 2024-03-29T11:29:22.734 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.727+0000 7feeeccd7640 1 -- 172.21.15.104:0/2428176566 wait complete. 2024-03-29T11:29:22.734 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.728+0000 7feeeccd7640 10 monclient: build_initial_monmap 2024-03-29T11:29:22.734 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.728+0000 7feeeccd7640 10 monclient: monmap: 2024-03-29T11:29:22.734 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:29:22.734 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:22.734 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:29:22.734 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:29:22.734 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:22.734 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:22.734 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:29:22.734 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:29:22.735 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:29:22.735 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:22.735 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.728+0000 7feeeccd7640 1 librados: starting msgr at 2024-03-29T11:29:22.735 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.728+0000 7feeeccd7640 1 librados: starting objecter 2024-03-29T11:29:22.735 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.728+0000 7feeeccd7640 1 Processor -- start 2024-03-29T11:29:22.735 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.728+0000 7feeeccd7640 1 -- start start 2024-03-29T11:29:22.735 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.728+0000 7feeeccd7640 1 librados: setting wanted keys 2024-03-29T11:29:22.735 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.728+0000 7feeeccd7640 1 librados: calling monclient init 2024-03-29T11:29:22.735 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.728+0000 7feeeccd7640 10 monclient: init 2024-03-29T11:29:22.735 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.729+0000 7feeeccd7640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:22.735 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.729+0000 7feeeccd7640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:29:22.735 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.729+0000 7feeeccd7640 10 monclient(hunting): picked mon.noname-c con 0x7feee0030660 addr v1:172.21.15.104:6790/0 2024-03-29T11:29:22.735 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.729+0000 7feeeccd7640 10 monclient(hunting): picked mon.noname-b con 0x7feee00389e0 addr v1:172.21.15.119:6789/0 2024-03-29T11:29:22.735 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.729+0000 7feeeccd7640 10 monclient(hunting): picked mon.noname-a con 0x7feee002f4b0 addr v1:172.21.15.104:6789/0 2024-03-29T11:29:22.736 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.729+0000 7feeeccd7640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feee00c7270 con 0x7feee002f4b0 2024-03-29T11:29:22.736 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.729+0000 7feeeccd7640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feee00c8160 con 0x7feee00389e0 2024-03-29T11:29:22.736 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.729+0000 7feeeccd7640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feee016e4e0 con 0x7feee0030660 2024-03-29T11:29:22.736 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.729+0000 7feeeccd7640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:22.736 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.729+0000 7feeeccd7640 10 monclient(hunting): authenticate will time out at 1543.980583s 2024-03-29T11:29:22.736 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.729+0000 7feee77fe640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7feee0030660 0x7feee00c7a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:58452/0 (socket says 172.21.15.104:58452) 2024-03-29T11:29:22.736 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.729+0000 7feee77fe640 1 -- 172.21.15.104:0/353861358 learned_addr learned my addr 172.21.15.104:0/353861358 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:29:22.736 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.730+0000 7feed3fff640 10 client.?.objecter ms_handle_connect 0x7feee0030660 2024-03-29T11:29:22.736 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.730+0000 7feed3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:22.736 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.730+0000 7feed3fff640 1 -- 172.21.15.104:0/353861358 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3285130795 0 0) 0x7feee016e4e0 con 0x7feee0030660 2024-03-29T11:29:22.736 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.730+0000 7feed3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:29:22.736 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.730+0000 7feed3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:22.736 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.730+0000 7feed3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:22.736 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.730+0000 7feed3fff640 10 monclient(hunting): my global_id is 4619 2024-03-29T11:29:22.736 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.730+0000 7feed3fff640 1 -- 172.21.15.104:0/353861358 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feecc004eb0 con 0x7feee0030660 2024-03-29T11:29:22.737 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.731+0000 7feed3fff640 1 -- 172.21.15.104:0/353861358 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1598729191 0 0) 0x7feecc004eb0 con 0x7feee0030660 2024-03-29T11:29:22.737 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.731+0000 7feed3fff640 1 -- 172.21.15.104:0/353861358 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feee016e4e0 con 0x7feee0030660 2024-03-29T11:29:22.737 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.731+0000 7feed3fff640 1 -- 172.21.15.104:0/353861358 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7feedc001d40 con 0x7feee0030660 2024-03-29T11:29:22.737 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.731+0000 7feed3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:22.737 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.731+0000 7feed3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:22.737 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.731+0000 7feed3fff640 10 monclient(hunting): dump: 2024-03-29T11:29:22.737 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:29:22.737 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:29:22.737 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:22.737 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:22.737 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:22.737 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:22.737 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:29:22.737 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:29:22.738 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:29:22.738 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:22.738 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.731+0000 7feed3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:22.738 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.731+0000 7feed3fff640 1 -- 172.21.15.104:0/353861358 <== mon.2 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1363949284 0 0) 0x7feee016e4e0 con 0x7feee0030660 2024-03-29T11:29:22.738 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.731+0000 7feed3fff640 1 -- 172.21.15.104:0/353861358 >> v1:172.21.15.119:6789/0 conn(0x7feee00389e0 legacy=0x7feee0160c20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:29:22.738 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.731+0000 7feed3fff640 1 -- 172.21.15.104:0/353861358 >> v1:172.21.15.104:6789/0 conn(0x7feee002f4b0 legacy=0x7feee0161330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:29:22.738 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.732+0000 7feed3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:29:22.738 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.732+0000 7feed3fff640 1 monclient: found mon.c 2024-03-29T11:29:22.738 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.732+0000 7feed3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:22.738 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.732+0000 7feed3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:29:22.738 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.732+0000 7feed3fff640 1 -- 172.21.15.104:0/353861358 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee016f680 con 0x7feee0030660 2024-03-29T11:29:22.738 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.732+0000 7feed3fff640 10 monclient: _finish_auth 0 2024-03-29T11:29:22.738 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.732+0000 7feed3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:29:22.738 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.732+0000 7feed3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:22.739 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.732+0000 7feeeccd7640 5 monclient: authenticate success, global_id 4619 2024-03-29T11:29:22.739 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.732+0000 7feeeccd7640 10 monclient: _renew_subs 2024-03-29T11:29:22.739 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.732+0000 7feeeccd7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:29:22.739 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.732+0000 7feeeccd7640 1 -- 172.21.15.104:0/353861358 --> v1:172.21.15.104:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feee016e670 con 0x7feee0030660 2024-03-29T11:29:22.739 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.732+0000 7feeeccd7640 10 client.4619.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:22.739 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.732+0000 7feeeccd7640 10 monclient: _renew_subs 2024-03-29T11:29:22.739 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.732+0000 7feeeccd7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:29:22.739 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.732+0000 7feeeccd7640 1 -- 172.21.15.104:0/353861358 --> v1:172.21.15.104:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feee016eba0 con 0x7feee0030660 2024-03-29T11:29:22.739 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.732+0000 7feeeccd7640 1 librados: init done 2024-03-29T11:29:22.739 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.732+0000 7feed3fff640 1 -- 172.21.15.104:0/353861358 <== mon.2 v1:172.21.15.104:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feedc003b30 con 0x7feee0030660 2024-03-29T11:29:22.739 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.732+0000 7feed3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:22.739 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.732+0000 7feed3fff640 1 -- 172.21.15.104:0/353861358 <== mon.2 v1:172.21.15.104:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7feedc003f40 con 0x7feee0030660 2024-03-29T11:29:22.739 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.732+0000 7feed3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:22.739 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.732+0000 7feed3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:22.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.732+0000 7feed3fff640 10 monclient: dump: 2024-03-29T11:29:22.740 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:29:22.740 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:29:22.740 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:22.740 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:22.740 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:22.740 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:22.740 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:29:22.740 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:29:22.740 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:29:22.740 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:22.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.732+0000 7feee5ffb640 4 set_mon_vals no callback set 2024-03-29T11:29:22.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.734+0000 7feeeccd7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:22.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.734+0000 7feeeccd7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:22.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.734+0000 7feeeccd7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:29:22.741 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.734+0000 7feeeccd7640 1 -- 172.21.15.104:0/353861358 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feec8005080 con 0x7feee0030660 2024-03-29T11:29:22.741 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.734+0000 7feed3fff640 1 -- 172.21.15.104:0/353861358 <== mon.2 v1:172.21.15.104:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7feedc0040a0 con 0x7feee0030660 2024-03-29T11:29:22.741 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.735+0000 7feed3fff640 1 -- 172.21.15.104:0/353861358 <== mon.2 v1:172.21.15.104:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2315086799 0 0) 0x7feedc084db0 con 0x7feee0030660 2024-03-29T11:29:22.741 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.735+0000 7feed3fff640 10 client.4619.objecter ms_dispatch 0x7feee8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:22.741 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.735+0000 7feed3fff640 3 client.4619.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:22.741 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.735+0000 7feed3fff640 3 client.4619.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:22.741 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.735+0000 7feed3fff640 20 client.4619.objecter dump_active .. 0 homeless 2024-03-29T11:29:22.741 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.735+0000 7feee6ffd640 10 monclient: get_auth_request con 0x7feecc06dad0 auth_method 0 2024-03-29T11:29:22.741 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.735+0000 7feed3fff640 10 client.4619.objecter ms_handle_connect 0x7feecc06dad0 2024-03-29T11:29:22.742 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.743+0000 7feed3fff640 1 -- 172.21.15.104:0/353861358 <== mon.2 v1:172.21.15.104:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7feedc0506f0 con 0x7feee0030660 2024-03-29T11:29:22.742 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.743+0000 7feed3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:22.742 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:22.743+0000 7feed3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:23.046 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:23.055+0000 7feeeccd7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:23.046 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:23.055+0000 7feeeccd7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:23.046 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:23.055+0000 7feeeccd7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:29:23.046 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:23.055+0000 7feeeccd7640 1 -- 172.21.15.104:0/353861358 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feec8005330 con 0x7feee0030660 2024-03-29T11:29:23.047 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:23.056+0000 7feed3fff640 1 -- 172.21.15.104:0/353861358 <== mon.2 v1:172.21.15.104:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 965847595) 0x7feedc084460 con 0x7feee0030660 2024-03-29T11:29:23.047 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:23.056+0000 7feed3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:23.047 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:23.056+0000 7feed3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:23.047 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:29:23.047 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":10,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:19:26.635370+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:26.635370+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:23.051 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:23.059+0000 7feeeccd7640 10 librados: watch_flush enter 2024-03-29T11:29:23.051 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:23.059+0000 7feeeccd7640 10 librados: watch_flush exit 2024-03-29T11:29:23.051 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:23.059+0000 7feeeccd7640 20 client.4619.objecter shutdown clearing up homeless session... 2024-03-29T11:29:23.051 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:23.060+0000 7feeeccd7640 10 client.4619.objecter successfully canceled tick 2024-03-29T11:29:23.051 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:23.060+0000 7feeeccd7640 1 -- 172.21.15.104:0/353861358 >> v1:172.21.15.104:6816/3341126660 conn(0x7feecc06dad0 legacy=0x7feecc06ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:23.051 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:23.060+0000 7feeeccd7640 10 monclient: shutdown 2024-03-29T11:29:23.051 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:23.060+0000 7feeeccd7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:23.052 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:23.060+0000 7feeeccd7640 1 -- 172.21.15.104:0/353861358 >> v1:172.21.15.104:6790/0 conn(0x7feee0030660 legacy=0x7feee00c7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:23.052 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:23.060+0000 7feeeccd7640 1 -- 172.21.15.104:0/353861358 shutdown_connections 2024-03-29T11:29:23.052 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:23.060+0000 7feeeccd7640 1 -- 172.21.15.104:0/353861358 >> 172.21.15.104:0/353861358 conn(0x7feee0005c00 msgr2=0x7feee0034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:23.052 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:23.060+0000 7feeeccd7640 1 -- 172.21.15.104:0/353861358 shutdown_connections 2024-03-29T11:29:23.052 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:23.060+0000 7feeeccd7640 1 -- 172.21.15.104:0/353861358 wait complete. 2024-03-29T11:29:23.052 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:23.060+0000 7feeeccd7640 1 librados: shutdown 2024-03-29T11:29:23.066 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:29.873 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:30.017 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.026+0000 7f170c082640 10 monclient: get_monmap_and_config 2024-03-29T11:29:30.018 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.026+0000 7f170c082640 10 monclient: build_initial_monmap 2024-03-29T11:29:30.018 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.026+0000 7f170c082640 10 monclient: monmap: 2024-03-29T11:29:30.018 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:29:30.018 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:29:30.018 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:29:30.027543+0000 2024-03-29T11:29:30.018 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:29:30.027543+0000 2024-03-29T11:29:30.018 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:30.018 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:30.018 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:29:30.018 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:29:30.018 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:29:30.019 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:30.019 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.027+0000 7f170c082640 1 Processor -- start 2024-03-29T11:29:30.020 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.027+0000 7f170c082640 1 -- start start 2024-03-29T11:29:30.020 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.027+0000 7f170c082640 10 monclient: init 2024-03-29T11:29:30.020 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.028+0000 7f170c082640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:30.020 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.028+0000 7f170c082640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:29:30.020 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.028+0000 7f170c082640 10 monclient(hunting): picked mon.noname-b con 0x7f17040b0770 addr v1:172.21.15.104:6790/0 2024-03-29T11:29:30.020 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.028+0000 7f170c082640 10 monclient(hunting): picked mon.noname-c con 0x7f170414a700 addr v1:172.21.15.119:6789/0 2024-03-29T11:29:30.020 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.028+0000 7f170c082640 10 monclient(hunting): picked mon.noname-a con 0x7f17040ac5a0 addr v1:172.21.15.104:6789/0 2024-03-29T11:29:30.020 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.028+0000 7f170c082640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17040aba70 con 0x7f17040ac5a0 2024-03-29T11:29:30.020 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.028+0000 7f170c082640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17040aecb0 con 0x7f170414a700 2024-03-29T11:29:30.020 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.028+0000 7f170c082640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17040af150 con 0x7f17040b0770 2024-03-29T11:29:30.020 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.028+0000 7f170c082640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:30.020 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.028+0000 7f170c082640 10 monclient(hunting): authenticate will time out at 1551.279730s 2024-03-29T11:29:30.021 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.030+0000 7f170961d640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f170414a700 0x7f17040abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:57018/0 (socket says 172.21.15.104:57018) 2024-03-29T11:29:30.021 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.030+0000 7f170961d640 1 -- 172.21.15.104:0/2141679256 learned_addr learned my addr 172.21.15.104:0/2141679256 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:29:30.022 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.031+0000 7f1708e1c640 1 -- 172.21.15.104:0/2141679256 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3211667628 0 0) 0x7f17040aecb0 con 0x7f170414a700 2024-03-29T11:29:30.022 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.031+0000 7f1708e1c640 10 monclient(hunting): _negotiate 2024-03-29T11:29:30.023 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.031+0000 7f1708e1c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:30.023 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.031+0000 7f1708e1c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:30.023 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.031+0000 7f1708e1c640 10 monclient(hunting): my global_id is 4567 2024-03-29T11:29:30.023 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.031+0000 7f1708e1c640 1 -- 172.21.15.104:0/2141679256 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16f0003520 con 0x7f170414a700 2024-03-29T11:29:30.023 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.031+0000 7f1708e1c640 1 -- 172.21.15.104:0/2141679256 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1998745178 0 0) 0x7f16f0003520 con 0x7f170414a700 2024-03-29T11:29:30.023 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.031+0000 7f1708e1c640 1 -- 172.21.15.104:0/2141679256 >> v1:172.21.15.104:6790/0 conn(0x7f17040b0770 legacy=0x7f1704149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:30.023 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.031+0000 7f1708e1c640 1 -- 172.21.15.104:0/2141679256 >> v1:172.21.15.104:6789/0 conn(0x7f17040ac5a0 legacy=0x7f1704151f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:30.024 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.032+0000 7f1708e1c640 10 monclient: _finish_hunting 0 2024-03-29T11:29:30.024 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.032+0000 7f1708e1c640 1 monclient: found mon.noname-c 2024-03-29T11:29:30.025 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.032+0000 7f1708e1c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:30.025 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.032+0000 7f1708e1c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.119:6789/0 2024-03-29T11:29:30.025 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.032+0000 7f1708e1c640 1 -- 172.21.15.104:0/2141679256 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17040ac9d0 con 0x7f170414a700 2024-03-29T11:29:30.025 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.032+0000 7f1708e1c640 10 monclient: _finish_auth 0 2024-03-29T11:29:30.025 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.032+0000 7f1708e1c640 10 monclient: _check_auth_tickets 2024-03-29T11:29:30.025 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.032+0000 7f1708e1c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:30.025 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.032+0000 7f170c082640 5 monclient: authenticate success, global_id 4567 2024-03-29T11:29:30.025 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.032+0000 7f170c082640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:30.025 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.032+0000 7f1708e1c640 1 -- 172.21.15.104:0/2141679256 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f16fc001a30 con 0x7f170414a700 2024-03-29T11:29:30.026 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.032+0000 7f1708e1c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:30.026 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.032+0000 7f1708e1c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:30.026 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.032+0000 7f1708e1c640 10 monclient: dump: 2024-03-29T11:29:30.026 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:29:30.026 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:29:30.026 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:30.026 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:30.026 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:30.026 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:30.027 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:29:30.027 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:29:30.027 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:29:30.027 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:30.027 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.032+0000 7f170c082640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:30.029 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.035+0000 7f1708e1c640 1 -- 172.21.15.104:0/2141679256 <== mon.1 v1:172.21.15.119:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f16fc001c90 con 0x7f170414a700 2024-03-29T11:29:30.029 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.036+0000 7f1708e1c640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:30.029 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.036+0000 7f170c082640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:30.029 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.036+0000 7f170c082640 4 set_mon_vals no callback set 2024-03-29T11:29:30.029 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.036+0000 7f1708e1c640 1 -- 172.21.15.104:0/2141679256 <== mon.1 v1:172.21.15.119:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f16fc0020e0 con 0x7f170414a700 2024-03-29T11:29:30.029 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.036+0000 7f1708e1c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:30.029 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.036+0000 7f1708e1c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:30.029 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.036+0000 7f1708e1c640 10 monclient: dump: 2024-03-29T11:29:30.029 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:29:30.029 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:29:30.029 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:30.030 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:30.030 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:30.030 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:30.030 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:29:30.030 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:29:30.030 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:29:30.030 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:30.030 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.036+0000 7f170c082640 10 monclient: shutdown 2024-03-29T11:29:30.030 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.036+0000 7f170c082640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:30.030 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.036+0000 7f170c082640 1 -- 172.21.15.104:0/2141679256 >> v1:172.21.15.119:6789/0 conn(0x7f170414a700 legacy=0x7f17040abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:30.030 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.037+0000 7f170c082640 1 -- 172.21.15.104:0/2141679256 shutdown_connections 2024-03-29T11:29:30.030 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.037+0000 7f170c082640 1 -- 172.21.15.104:0/2141679256 >> 172.21.15.104:0/2141679256 conn(0x7f170406b660 msgr2=0x7f170406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:30.030 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.037+0000 7f170c082640 1 -- 172.21.15.104:0/2141679256 shutdown_connections 2024-03-29T11:29:30.031 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.037+0000 7f170c082640 1 -- 172.21.15.104:0/2141679256 wait complete. 2024-03-29T11:29:30.032 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.038+0000 7f170c082640 10 monclient: build_initial_monmap 2024-03-29T11:29:30.032 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.038+0000 7f170c082640 10 monclient: monmap: 2024-03-29T11:29:30.032 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:29:30.032 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:30.032 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:29:30.032 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:29:30.032 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:30.032 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:30.032 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:29:30.032 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:29:30.032 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:29:30.032 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:30.032 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.038+0000 7f170c082640 1 librados: starting msgr at 2024-03-29T11:29:30.032 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.038+0000 7f170c082640 1 librados: starting objecter 2024-03-29T11:29:30.033 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.038+0000 7f170c082640 1 Processor -- start 2024-03-29T11:29:30.033 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.039+0000 7f170c082640 1 -- start start 2024-03-29T11:29:30.033 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.039+0000 7f170c082640 1 librados: setting wanted keys 2024-03-29T11:29:30.033 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.039+0000 7f170c082640 1 librados: calling monclient init 2024-03-29T11:29:30.035 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.039+0000 7f170c082640 10 monclient: init 2024-03-29T11:29:30.035 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.040+0000 7f170c082640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:30.035 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.040+0000 7f170c082640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:29:30.035 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.040+0000 7f170c082640 10 monclient(hunting): picked mon.noname-c con 0x7f17040ac5a0 addr v1:172.21.15.104:6790/0 2024-03-29T11:29:30.035 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.040+0000 7f170c082640 10 monclient(hunting): picked mon.noname-a con 0x7f17040b0770 addr v1:172.21.15.104:6789/0 2024-03-29T11:29:30.035 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.040+0000 7f170c082640 10 monclient(hunting): picked mon.noname-b con 0x7f170414a700 addr v1:172.21.15.119:6789/0 2024-03-29T11:29:30.035 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.040+0000 7f170c082640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1704120360 con 0x7f17040b0770 2024-03-29T11:29:30.035 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.040+0000 7f170c082640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1704120640 con 0x7f170414a700 2024-03-29T11:29:30.036 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.040+0000 7f170c082640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f170411c610 con 0x7f17040ac5a0 2024-03-29T11:29:30.036 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.040+0000 7f170c082640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:30.036 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.040+0000 7f170c082640 10 monclient(hunting): authenticate will time out at 1551.291226s 2024-03-29T11:29:30.036 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.041+0000 7f170a61f640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f170414a700 0x7f17041ee9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:57710/0 (socket says 172.21.15.104:57710) 2024-03-29T11:29:30.036 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.041+0000 7f170a61f640 1 -- 172.21.15.104:0/1577685627 learned_addr learned my addr 172.21.15.104:0/1577685627 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:29:30.036 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.041+0000 7f16fa7fc640 10 client.?.objecter ms_handle_connect 0x7f170414a700 2024-03-29T11:29:30.036 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.041+0000 7f16fa7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:30.036 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.043+0000 7f16fa7fc640 1 -- 172.21.15.104:0/1577685627 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1233160635 0 0) 0x7f1704120640 con 0x7f170414a700 2024-03-29T11:29:30.036 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.043+0000 7f16fa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:29:30.036 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.043+0000 7f16fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:30.036 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.043+0000 7f16fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:30.036 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.043+0000 7f16fa7fc640 10 monclient(hunting): my global_id is 4570 2024-03-29T11:29:30.036 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.043+0000 7f16fa7fc640 1 -- 172.21.15.104:0/1577685627 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16e8004eb0 con 0x7f170414a700 2024-03-29T11:29:30.036 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.043+0000 7f16fa7fc640 1 -- 172.21.15.104:0/1577685627 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3201385441 0 0) 0x7f16e8004eb0 con 0x7f170414a700 2024-03-29T11:29:30.037 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.044+0000 7f16fa7fc640 1 -- 172.21.15.104:0/1577685627 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1704120640 con 0x7f170414a700 2024-03-29T11:29:30.037 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.044+0000 7f16fa7fc640 1 -- 172.21.15.104:0/1577685627 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f17000018e0 con 0x7f170414a700 2024-03-29T11:29:30.037 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.044+0000 7f16fa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:30.037 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.044+0000 7f16fa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:30.037 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.044+0000 7f16fa7fc640 10 monclient(hunting): dump: 2024-03-29T11:29:30.037 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:29:30.037 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:29:30.037 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:30.037 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:30.037 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:30.037 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:30.037 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:29:30.037 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:29:30.037 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:29:30.037 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:30.038 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.044+0000 7f16fa7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:30.038 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.044+0000 7f16fa7fc640 10 client.?.objecter ms_handle_connect 0x7f17040ac5a0 2024-03-29T11:29:30.038 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.044+0000 7f16fa7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:30.038 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.044+0000 7f16fa7fc640 1 -- 172.21.15.104:0/1577685627 <== mon.1 v1:172.21.15.119:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2359668606 0 0) 0x7f1704120640 con 0x7f170414a700 2024-03-29T11:29:30.038 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.044+0000 7f16fa7fc640 1 -- 172.21.15.104:0/1577685627 >> v1:172.21.15.104:6790/0 conn(0x7f17040ac5a0 legacy=0x7f170411e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:30.038 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.044+0000 7f16fa7fc640 1 -- 172.21.15.104:0/1577685627 >> v1:172.21.15.104:6789/0 conn(0x7f17040b0770 legacy=0x7f170411e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:30.039 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.044+0000 7f16fa7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:29:30.040 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.044+0000 7f16fa7fc640 1 monclient: found mon.b 2024-03-29T11:29:30.040 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.044+0000 7f16fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:30.040 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.044+0000 7f16fa7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:29:30.040 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.044+0000 7f16fa7fc640 1 -- 172.21.15.104:0/1577685627 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1704123760 con 0x7f170414a700 2024-03-29T11:29:30.040 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.044+0000 7f16fa7fc640 10 monclient: _finish_auth 0 2024-03-29T11:29:30.040 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.044+0000 7f16fa7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:29:30.040 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.044+0000 7f16fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:30.040 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.044+0000 7f170c082640 5 monclient: authenticate success, global_id 4570 2024-03-29T11:29:30.040 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.044+0000 7f170c082640 10 monclient: _renew_subs 2024-03-29T11:29:30.040 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.044+0000 7f170c082640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:29:30.040 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.044+0000 7f170c082640 1 -- 172.21.15.104:0/1577685627 --> v1:172.21.15.119:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f170411f070 con 0x7f170414a700 2024-03-29T11:29:30.040 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.044+0000 7f170c082640 10 client.4570.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:30.041 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.044+0000 7f170c082640 10 monclient: _renew_subs 2024-03-29T11:29:30.041 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.044+0000 7f170c082640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:29:30.041 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.044+0000 7f170c082640 1 -- 172.21.15.104:0/1577685627 --> v1:172.21.15.119:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f17041474e0 con 0x7f170414a700 2024-03-29T11:29:30.041 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.044+0000 7f170c082640 1 librados: init done 2024-03-29T11:29:30.041 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.046+0000 7f170c082640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:30.041 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.046+0000 7f170c082640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:30.041 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.046+0000 7f170c082640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:29:30.041 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.046+0000 7f170c082640 1 -- 172.21.15.104:0/1577685627 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17040ab560 con 0x7f170414a700 2024-03-29T11:29:30.041 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.047+0000 7f16fa7fc640 1 -- 172.21.15.104:0/1577685627 <== mon.1 v1:172.21.15.119:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1700001d40 con 0x7f170414a700 2024-03-29T11:29:30.041 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.047+0000 7f16fa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:30.041 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.047+0000 7f16fa7fc640 1 -- 172.21.15.104:0/1577685627 <== mon.1 v1:172.21.15.119:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f1700001bc0 con 0x7f170414a700 2024-03-29T11:29:30.041 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.047+0000 7f16fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:30.041 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.047+0000 7f16fa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:30.041 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.047+0000 7f16fa7fc640 10 monclient: dump: 2024-03-29T11:29:30.042 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:29:30.042 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:29:30.042 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:30.042 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:30.042 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:30.042 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:30.042 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:29:30.042 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:29:30.042 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:29:30.042 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:30.042 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.048+0000 7f16fbfff640 4 set_mon_vals no callback set 2024-03-29T11:29:30.042 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.049+0000 7f16fa7fc640 1 -- 172.21.15.104:0/1577685627 <== mon.1 v1:172.21.15.119:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f1700003a40 con 0x7f170414a700 2024-03-29T11:29:30.043 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.049+0000 7f1709e1e640 10 monclient: get_auth_request con 0x7f16e806dbe0 auth_method 0 2024-03-29T11:29:30.043 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.050+0000 7f16fa7fc640 1 -- 172.21.15.104:0/1577685627 <== mon.1 v1:172.21.15.119:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 668040924 0 0) 0x7f1700084a40 con 0x7f170414a700 2024-03-29T11:29:30.043 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.050+0000 7f16fa7fc640 10 client.4570.objecter ms_dispatch 0x7f1704000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:29:30.044 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.050+0000 7f16fa7fc640 3 client.4570.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:29:30.044 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.050+0000 7f16fa7fc640 3 client.4570.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:29:30.044 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.050+0000 7f16fa7fc640 20 client.4570.objecter dump_active .. 0 homeless 2024-03-29T11:29:30.044 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.050+0000 7f16fa7fc640 10 client.4570.objecter ms_handle_connect 0x7f16e806dbe0 2024-03-29T11:29:30.050 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.059+0000 7f16fa7fc640 1 -- 172.21.15.104:0/1577685627 <== mon.1 v1:172.21.15.119:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1700050150 con 0x7f170414a700 2024-03-29T11:29:30.050 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.059+0000 7f16fa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:30.050 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.059+0000 7f16fa7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:30.365 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.374+0000 7f170c082640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:30.366 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.374+0000 7f170c082640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:30.366 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.374+0000 7f170c082640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.119:6789/0 2024-03-29T11:29:30.366 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.374+0000 7f170c082640 1 -- 172.21.15.104:0/1577685627 --> v1:172.21.15.119:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1704075f10 con 0x7f170414a700 2024-03-29T11:29:30.367 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.376+0000 7f16fa7fc640 1 -- 172.21.15.104:0/1577685627 <== mon.1 v1:172.21.15.119:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3673681244) 0x7f1700052250 con 0x7f170414a700 2024-03-29T11:29:30.367 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.376+0000 7f16fa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:30.368 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.376+0000 7f16fa7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:30.369 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:29:30.369 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":11,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:29:29.328850+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:29.328850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:30.372 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.380+0000 7f170c082640 10 librados: watch_flush enter 2024-03-29T11:29:30.372 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.380+0000 7f170c082640 10 librados: watch_flush exit 2024-03-29T11:29:30.372 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.380+0000 7f170c082640 20 client.4570.objecter shutdown clearing up homeless session... 2024-03-29T11:29:30.372 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.380+0000 7f170c082640 10 client.4570.objecter successfully canceled tick 2024-03-29T11:29:30.373 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.380+0000 7f170c082640 1 -- 172.21.15.104:0/1577685627 >> v1:172.21.15.104:6816/3341126660 conn(0x7f16e806dbe0 legacy=0x7f16e80700a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:30.374 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.380+0000 7f170c082640 10 monclient: shutdown 2024-03-29T11:29:30.374 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.381+0000 7f170c082640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:30.374 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.381+0000 7f170c082640 1 -- 172.21.15.104:0/1577685627 >> v1:172.21.15.119:6789/0 conn(0x7f170414a700 legacy=0x7f17041ee9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:30.374 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.381+0000 7f170c082640 1 -- 172.21.15.104:0/1577685627 shutdown_connections 2024-03-29T11:29:30.374 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.381+0000 7f170c082640 1 -- 172.21.15.104:0/1577685627 >> 172.21.15.104:0/1577685627 conn(0x7f170406b660 msgr2=0x7f17040b2e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:30.374 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.381+0000 7f170c082640 1 -- 172.21.15.104:0/1577685627 shutdown_connections 2024-03-29T11:29:30.374 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.381+0000 7f170c082640 1 -- 172.21.15.104:0/1577685627 wait complete. 2024-03-29T11:29:30.374 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:30.381+0000 7f170c082640 1 librados: shutdown 2024-03-29T11:29:30.389 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:37.197 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:37.351 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.360+0000 7f00d0c96640 10 monclient: get_monmap_and_config 2024-03-29T11:29:37.352 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.360+0000 7f00d0c96640 10 monclient: build_initial_monmap 2024-03-29T11:29:37.352 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.360+0000 7f00d0c96640 10 monclient: monmap: 2024-03-29T11:29:37.352 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:29:37.352 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:29:37.352 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:29:37.361068+0000 2024-03-29T11:29:37.352 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:29:37.361068+0000 2024-03-29T11:29:37.352 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:37.352 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:37.352 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:29:37.352 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:29:37.352 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:29:37.352 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:37.355 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.361+0000 7f00d0c96640 1 Processor -- start 2024-03-29T11:29:37.355 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.362+0000 7f00d0c96640 1 -- start start 2024-03-29T11:29:37.355 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.362+0000 7f00d0c96640 10 monclient: init 2024-03-29T11:29:37.355 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.363+0000 7f00d0c96640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:37.355 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.363+0000 7f00d0c96640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:29:37.355 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.363+0000 7f00d0c96640 10 monclient(hunting): picked mon.noname-a con 0x7f00cc0b0860 addr v1:172.21.15.104:6789/0 2024-03-29T11:29:37.355 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.363+0000 7f00d0c96640 10 monclient(hunting): picked mon.noname-b con 0x7f00cc13ef30 addr v1:172.21.15.104:6790/0 2024-03-29T11:29:37.355 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.363+0000 7f00d0c96640 10 monclient(hunting): picked mon.noname-c con 0x7f00cc0af8e0 addr v1:172.21.15.119:6789/0 2024-03-29T11:29:37.355 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.363+0000 7f00d0c96640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00cc0b0c40 con 0x7f00cc0b0860 2024-03-29T11:29:37.356 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.363+0000 7f00d0c96640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00cc0aed10 con 0x7f00cc0af8e0 2024-03-29T11:29:37.356 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.363+0000 7f00d0c96640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00cc0afd50 con 0x7f00cc13ef30 2024-03-29T11:29:37.356 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.363+0000 7f00d0c96640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:37.356 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.363+0000 7f00d0c96640 10 monclient(hunting): authenticate will time out at 1558.614186s 2024-03-29T11:29:37.359 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.367+0000 7f00ca59c640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f00cc0b0860 0x7f00cc13e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:39043/0 (socket says 172.21.15.104:39043) 2024-03-29T11:29:37.359 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.367+0000 7f00ca59c640 1 -- 172.21.15.104:0/808862903 learned_addr learned my addr 172.21.15.104:0/808862903 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:29:37.363 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.368+0000 7f00c959a640 1 -- 172.21.15.104:0/808862903 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 309175991 0 0) 0x7f00cc0aed10 con 0x7f00cc0af8e0 2024-03-29T11:29:37.363 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.368+0000 7f00c959a640 10 monclient(hunting): _negotiate 2024-03-29T11:29:37.363 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.368+0000 7f00c959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:37.363 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.368+0000 7f00c959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:37.363 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.368+0000 7f00c959a640 10 monclient(hunting): my global_id is 4573 2024-03-29T11:29:37.363 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.368+0000 7f00c959a640 1 -- 172.21.15.104:0/808862903 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00b4003520 con 0x7f00cc0af8e0 2024-03-29T11:29:37.364 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.371+0000 7f00c959a640 1 -- 172.21.15.104:0/808862903 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3159389833 0 0) 0x7f00cc0b0c40 con 0x7f00cc0b0860 2024-03-29T11:29:37.364 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.371+0000 7f00c959a640 10 monclient(hunting): _negotiate 2024-03-29T11:29:37.364 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.371+0000 7f00c959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:37.364 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.371+0000 7f00c959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:37.364 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.371+0000 7f00c959a640 10 monclient(hunting): my global_id is 4569 2024-03-29T11:29:37.364 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.371+0000 7f00c959a640 1 -- 172.21.15.104:0/808862903 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00cc0aed10 con 0x7f00cc0b0860 2024-03-29T11:29:37.364 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.371+0000 7f00c959a640 1 -- 172.21.15.104:0/808862903 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 735203818 0 0) 0x7f00b4003520 con 0x7f00cc0af8e0 2024-03-29T11:29:37.364 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.371+0000 7f00c959a640 1 -- 172.21.15.104:0/808862903 >> v1:172.21.15.104:6790/0 conn(0x7f00cc13ef30 legacy=0x7f00cc0af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:37.364 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.371+0000 7f00c959a640 1 -- 172.21.15.104:0/808862903 >> v1:172.21.15.104:6789/0 conn(0x7f00cc0b0860 legacy=0x7f00cc13e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:37.364 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.371+0000 7f00c959a640 10 monclient: _finish_hunting 0 2024-03-29T11:29:37.368 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.371+0000 7f00c959a640 1 monclient: found mon.noname-c 2024-03-29T11:29:37.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.371+0000 7f00c959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:37.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.371+0000 7f00c959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.119:6789/0 2024-03-29T11:29:37.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.371+0000 7f00c959a640 1 -- 172.21.15.104:0/808862903 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00cc0ab1b0 con 0x7f00cc0af8e0 2024-03-29T11:29:37.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.371+0000 7f00c959a640 10 monclient: _finish_auth 0 2024-03-29T11:29:37.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.371+0000 7f00c959a640 10 monclient: _check_auth_tickets 2024-03-29T11:29:37.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.371+0000 7f00c959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:37.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.371+0000 7f00c959a640 1 -- 172.21.15.104:0/808862903 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f00c0001690 con 0x7f00cc0af8e0 2024-03-29T11:29:37.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.371+0000 7f00c959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:37.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.371+0000 7f00c959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:37.369 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.371+0000 7f00c959a640 10 monclient: dump: 2024-03-29T11:29:37.369 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:29:37.369 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:29:37.369 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:37.370 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:37.370 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:37.370 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:37.370 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:29:37.370 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:29:37.370 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:29:37.370 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:37.370 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.371+0000 7f00d0c96640 5 monclient: authenticate success, global_id 4573 2024-03-29T11:29:37.370 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.371+0000 7f00d0c96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:37.370 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.376+0000 7f00c959a640 1 -- 172.21.15.104:0/808862903 <== mon.1 v1:172.21.15.119:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00c00017f0 con 0x7f00cc0af8e0 2024-03-29T11:29:37.370 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.376+0000 7f00c959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:37.370 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.376+0000 7f00c959a640 1 -- 172.21.15.104:0/808862903 <== mon.1 v1:172.21.15.119:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f00c0001c00 con 0x7f00cc0af8e0 2024-03-29T11:29:37.370 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.376+0000 7f00c959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:37.370 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.376+0000 7f00c959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:37.371 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.376+0000 7f00c959a640 10 monclient: dump: 2024-03-29T11:29:37.371 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:29:37.371 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:29:37.371 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:37.371 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:37.371 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:37.371 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:37.371 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:29:37.371 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:29:37.371 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:29:37.371 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:37.371 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.376+0000 7f00d0c96640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:37.371 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.376+0000 7f00d0c96640 4 set_mon_vals no callback set 2024-03-29T11:29:37.371 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.377+0000 7f00d0c96640 10 monclient: shutdown 2024-03-29T11:29:37.371 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.377+0000 7f00d0c96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:37.371 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.377+0000 7f00d0c96640 1 -- 172.21.15.104:0/808862903 >> v1:172.21.15.119:6789/0 conn(0x7f00cc0af8e0 legacy=0x7f00cc149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:37.372 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.377+0000 7f00d0c96640 1 -- 172.21.15.104:0/808862903 shutdown_connections 2024-03-29T11:29:37.372 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.377+0000 7f00d0c96640 1 -- 172.21.15.104:0/808862903 >> 172.21.15.104:0/808862903 conn(0x7f00cc06b660 msgr2=0x7f00cc06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:37.372 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.380+0000 7f00d0c96640 1 -- 172.21.15.104:0/808862903 shutdown_connections 2024-03-29T11:29:37.372 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.380+0000 7f00d0c96640 1 -- 172.21.15.104:0/808862903 wait complete. 2024-03-29T11:29:37.373 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.381+0000 7f00d0c96640 10 monclient: build_initial_monmap 2024-03-29T11:29:37.373 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.381+0000 7f00d0c96640 10 monclient: monmap: 2024-03-29T11:29:37.373 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:29:37.374 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:37.374 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:29:37.374 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:29:37.374 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:37.375 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:37.375 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:29:37.375 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:29:37.375 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:29:37.375 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:37.375 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.382+0000 7f00d0c96640 1 librados: starting msgr at 2024-03-29T11:29:37.375 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.382+0000 7f00d0c96640 1 librados: starting objecter 2024-03-29T11:29:37.375 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.382+0000 7f00d0c96640 1 Processor -- start 2024-03-29T11:29:37.377 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.384+0000 7f00d0c96640 1 -- start start 2024-03-29T11:29:37.377 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.384+0000 7f00d0c96640 1 librados: setting wanted keys 2024-03-29T11:29:37.377 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.384+0000 7f00d0c96640 1 librados: calling monclient init 2024-03-29T11:29:37.377 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.384+0000 7f00d0c96640 10 monclient: init 2024-03-29T11:29:37.377 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.385+0000 7f00d0c96640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:37.377 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.385+0000 7f00d0c96640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:29:37.377 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.385+0000 7f00d0c96640 10 monclient(hunting): picked mon.noname-c con 0x7f00cc0af8e0 addr v1:172.21.15.104:6790/0 2024-03-29T11:29:37.377 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.385+0000 7f00d0c96640 10 monclient(hunting): picked mon.noname-b con 0x7f00cc0b0860 addr v1:172.21.15.119:6789/0 2024-03-29T11:29:37.383 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.389+0000 7f00d0c96640 10 monclient(hunting): picked mon.noname-a con 0x7f00cc13ef30 addr v1:172.21.15.104:6789/0 2024-03-29T11:29:37.383 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.389+0000 7f00d0c96640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00cc138630 con 0x7f00cc13ef30 2024-03-29T11:29:37.383 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.389+0000 7f00d0c96640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00cc1dc920 con 0x7f00cc0b0860 2024-03-29T11:29:37.384 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.389+0000 7f00d0c96640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00cc0b24a0 con 0x7f00cc0af8e0 2024-03-29T11:29:37.384 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.389+0000 7f00d0c96640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:37.384 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.389+0000 7f00d0c96640 10 monclient(hunting): authenticate will time out at 1558.640631s 2024-03-29T11:29:37.384 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.389+0000 7f00cad9d640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f00cc13ef30 0x7f00cc1dc210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:40001/0 (socket says 172.21.15.104:40001) 2024-03-29T11:29:37.384 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.389+0000 7f00cad9d640 1 -- 172.21.15.104:0/1834918258 learned_addr learned my addr 172.21.15.104:0/1834918258 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:29:37.384 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.390+0000 7f00baffd640 10 client.?.objecter ms_handle_connect 0x7f00cc13ef30 2024-03-29T11:29:37.384 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.390+0000 7f00baffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:37.384 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.390+0000 7f00baffd640 10 client.?.objecter ms_handle_connect 0x7f00cc0b0860 2024-03-29T11:29:37.384 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.390+0000 7f00baffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:37.384 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.390+0000 7f00baffd640 1 -- 172.21.15.104:0/1834918258 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2993854124 0 0) 0x7f00cc138630 con 0x7f00cc13ef30 2024-03-29T11:29:37.384 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.390+0000 7f00baffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:37.384 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.390+0000 7f00baffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:37.385 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.390+0000 7f00baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:37.385 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.390+0000 7f00baffd640 10 monclient(hunting): my global_id is 4572 2024-03-29T11:29:37.385 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.390+0000 7f00baffd640 1 -- 172.21.15.104:0/1834918258 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00ac004eb0 con 0x7f00cc13ef30 2024-03-29T11:29:37.385 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.390+0000 7f00baffd640 1 -- 172.21.15.104:0/1834918258 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1692166978 0 0) 0x7f00cc1dc920 con 0x7f00cc0b0860 2024-03-29T11:29:37.385 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.390+0000 7f00baffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:37.385 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.390+0000 7f00baffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:37.385 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.390+0000 7f00baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:37.385 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.390+0000 7f00baffd640 10 monclient(hunting): my global_id is 4576 2024-03-29T11:29:37.385 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.390+0000 7f00baffd640 1 -- 172.21.15.104:0/1834918258 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00cc138630 con 0x7f00cc0b0860 2024-03-29T11:29:37.385 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.390+0000 7f00baffd640 1 -- 172.21.15.104:0/1834918258 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4261805469 0 0) 0x7f00ac004eb0 con 0x7f00cc13ef30 2024-03-29T11:29:37.386 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.391+0000 7f00baffd640 1 -- 172.21.15.104:0/1834918258 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00cc1dc920 con 0x7f00cc13ef30 2024-03-29T11:29:37.386 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.391+0000 7f00baffd640 1 -- 172.21.15.104:0/1834918258 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f00c0001b40 con 0x7f00cc13ef30 2024-03-29T11:29:37.386 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.391+0000 7f00baffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:37.386 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.391+0000 7f00baffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:37.386 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.391+0000 7f00baffd640 10 monclient(hunting): dump: 2024-03-29T11:29:37.386 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:29:37.386 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:29:37.386 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:37.386 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:37.387 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:37.387 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:37.387 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:29:37.387 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:29:37.387 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:29:37.387 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:37.387 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.391+0000 7f00baffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:37.387 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.391+0000 7f00baffd640 1 -- 172.21.15.104:0/1834918258 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 518959501 0 0) 0x7f00cc138630 con 0x7f00cc0b0860 2024-03-29T11:29:37.387 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.391+0000 7f00baffd640 1 -- 172.21.15.104:0/1834918258 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00ac004eb0 con 0x7f00cc0b0860 2024-03-29T11:29:37.387 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.391+0000 7f00baffd640 1 -- 172.21.15.104:0/1834918258 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f00bc001b90 con 0x7f00cc0b0860 2024-03-29T11:29:37.387 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.391+0000 7f00baffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:37.387 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.391+0000 7f00baffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:37.387 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.391+0000 7f00baffd640 10 monclient(hunting): dump: 2024-03-29T11:29:37.387 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:29:37.387 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:29:37.388 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:37.388 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:37.388 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:37.388 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:37.388 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:29:37.388 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:29:37.388 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:29:37.388 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:37.388 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.391+0000 7f00baffd640 1 -- 172.21.15.104:0/1834918258 <== mon.0 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1251525691 0 0) 0x7f00cc1dc920 con 0x7f00cc13ef30 2024-03-29T11:29:37.388 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.391+0000 7f00baffd640 1 -- 172.21.15.104:0/1834918258 >> v1:172.21.15.104:6790/0 conn(0x7f00cc0af8e0 legacy=0x7f00cc137f20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:29:37.388 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.393+0000 7f00baffd640 1 -- 172.21.15.104:0/1834918258 >> v1:172.21.15.119:6789/0 conn(0x7f00cc0b0860 legacy=0x7f00cc1dbb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:37.388 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.393+0000 7f00baffd640 10 monclient: _finish_hunting 0 2024-03-29T11:29:37.388 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.393+0000 7f00baffd640 1 monclient: found mon.a 2024-03-29T11:29:37.388 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.393+0000 7f00baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:37.388 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.393+0000 7f00baffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:29:37.389 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.393+0000 7f00baffd640 1 -- 172.21.15.104:0/1834918258 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00cc137980 con 0x7f00cc13ef30 2024-03-29T11:29:37.389 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.393+0000 7f00baffd640 10 monclient: _finish_auth 0 2024-03-29T11:29:37.389 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.393+0000 7f00baffd640 10 monclient: _check_auth_tickets 2024-03-29T11:29:37.389 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.393+0000 7f00baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:37.389 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.393+0000 7f00d0c96640 5 monclient: authenticate success, global_id 4572 2024-03-29T11:29:37.389 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.393+0000 7f00d0c96640 10 monclient: _renew_subs 2024-03-29T11:29:37.389 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.393+0000 7f00d0c96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:29:37.389 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.393+0000 7f00d0c96640 1 -- 172.21.15.104:0/1834918258 --> v1:172.21.15.104:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00cc0b11d0 con 0x7f00cc13ef30 2024-03-29T11:29:37.389 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.393+0000 7f00d0c96640 10 client.4572.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:37.389 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.393+0000 7f00d0c96640 10 monclient: _renew_subs 2024-03-29T11:29:37.389 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.393+0000 7f00d0c96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:29:37.389 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.393+0000 7f00d0c96640 1 -- 172.21.15.104:0/1834918258 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f00cc1375d0 con 0x7f00cc13ef30 2024-03-29T11:29:37.389 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.393+0000 7f00d0c96640 1 librados: init done 2024-03-29T11:29:37.390 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.395+0000 7f00d0c96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:37.391 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.395+0000 7f00d0c96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:37.391 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.395+0000 7f00d0c96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:29:37.391 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.395+0000 7f00d0c96640 1 -- 172.21.15.104:0/1834918258 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00cc075f10 con 0x7f00cc13ef30 2024-03-29T11:29:37.391 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.396+0000 7f00baffd640 1 -- 172.21.15.104:0/1834918258 <== mon.0 v1:172.21.15.104:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00c0001ca0 con 0x7f00cc13ef30 2024-03-29T11:29:37.391 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.396+0000 7f00baffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:37.391 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.396+0000 7f00baffd640 1 -- 172.21.15.104:0/1834918258 <== mon.0 v1:172.21.15.104:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f00c0003a40 con 0x7f00cc13ef30 2024-03-29T11:29:37.391 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.396+0000 7f00baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:37.391 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.396+0000 7f00baffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:37.391 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.396+0000 7f00baffd640 10 monclient: dump: 2024-03-29T11:29:37.391 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:29:37.391 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:29:37.391 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:37.391 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:37.392 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:37.392 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:37.392 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:29:37.392 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:29:37.392 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:29:37.392 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:37.392 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.396+0000 7f00baffd640 1 -- 172.21.15.104:0/1834918258 <== mon.0 v1:172.21.15.104:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f00c0003c60 con 0x7f00cc13ef30 2024-03-29T11:29:37.392 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.397+0000 7f00baffd640 1 -- 172.21.15.104:0/1834918258 <== mon.0 v1:172.21.15.104:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 668040924 0 0) 0x7f00c0084bd0 con 0x7f00cc13ef30 2024-03-29T11:29:37.392 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.397+0000 7f00baffd640 10 client.4572.objecter ms_dispatch 0x7f00cc000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:29:37.392 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.397+0000 7f00baffd640 3 client.4572.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:29:37.392 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.397+0000 7f00baffd640 3 client.4572.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:29:37.392 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.397+0000 7f00baffd640 20 client.4572.objecter dump_active .. 0 homeless 2024-03-29T11:29:37.392 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.397+0000 7f00c8d99640 4 set_mon_vals no callback set 2024-03-29T11:29:37.392 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.398+0000 7f00ca59c640 10 monclient: get_auth_request con 0x7f00ac06de90 auth_method 0 2024-03-29T11:29:37.393 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.398+0000 7f00baffd640 10 client.4572.objecter ms_handle_connect 0x7f00ac06de90 2024-03-29T11:29:37.395 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.404+0000 7f00baffd640 1 -- 172.21.15.104:0/1834918258 <== mon.0 v1:172.21.15.104:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f00c0050530 con 0x7f00cc13ef30 2024-03-29T11:29:37.395 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.404+0000 7f00baffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:37.395 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.404+0000 7f00baffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:37.718 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.725+0000 7f00d0c96640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:37.718 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.725+0000 7f00d0c96640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:37.718 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.725+0000 7f00d0c96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:29:37.718 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.725+0000 7f00d0c96640 1 -- 172.21.15.104:0/1834918258 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f00cc072330 con 0x7f00cc13ef30 2024-03-29T11:29:37.719 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.726+0000 7f00baffd640 1 -- 172.21.15.104:0/1834918258 <== mon.0 v1:172.21.15.104:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3673681244) 0x7f00c0052460 con 0x7f00cc13ef30 2024-03-29T11:29:37.719 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.727+0000 7f00baffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:37.719 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.727+0000 7f00baffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:37.719 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:29:37.719 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":11,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:29:29.328850+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:29.328850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:37.723 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.731+0000 7f00d0c96640 10 librados: watch_flush enter 2024-03-29T11:29:37.723 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.731+0000 7f00d0c96640 10 librados: watch_flush exit 2024-03-29T11:29:37.723 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.731+0000 7f00d0c96640 20 client.4572.objecter shutdown clearing up homeless session... 2024-03-29T11:29:37.723 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.731+0000 7f00d0c96640 10 client.4572.objecter successfully canceled tick 2024-03-29T11:29:37.723 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.731+0000 7f00d0c96640 1 -- 172.21.15.104:0/1834918258 >> v1:172.21.15.104:6816/3341126660 conn(0x7f00ac06de90 legacy=0x7f00ac070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:37.723 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.731+0000 7f00d0c96640 10 monclient: shutdown 2024-03-29T11:29:37.724 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.731+0000 7f00d0c96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:37.724 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.731+0000 7f00d0c96640 1 -- 172.21.15.104:0/1834918258 >> v1:172.21.15.104:6789/0 conn(0x7f00cc13ef30 legacy=0x7f00cc1dc210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:37.724 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.731+0000 7f00d0c96640 1 -- 172.21.15.104:0/1834918258 shutdown_connections 2024-03-29T11:29:37.724 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.731+0000 7f00d0c96640 1 -- 172.21.15.104:0/1834918258 >> 172.21.15.104:0/1834918258 conn(0x7f00cc06b660 msgr2=0x7f00cc1c8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:37.724 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.732+0000 7f00d0c96640 1 -- 172.21.15.104:0/1834918258 shutdown_connections 2024-03-29T11:29:37.725 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.732+0000 7f00d0c96640 1 -- 172.21.15.104:0/1834918258 wait complete. 2024-03-29T11:29:37.725 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:37.733+0000 7f00d0c96640 1 librados: shutdown 2024-03-29T11:29:37.740 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:44.548 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:44.707 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.716+0000 7febb8da3640 10 monclient: get_monmap_and_config 2024-03-29T11:29:44.708 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.716+0000 7febb8da3640 10 monclient: build_initial_monmap 2024-03-29T11:29:44.708 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.716+0000 7febb8da3640 10 monclient: monmap: 2024-03-29T11:29:44.708 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:29:44.708 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:29:44.708 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:29:44.717065+0000 2024-03-29T11:29:44.708 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:29:44.717065+0000 2024-03-29T11:29:44.708 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:44.708 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:44.708 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:29:44.708 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:29:44.708 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:29:44.708 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:44.709 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.717+0000 7febb8da3640 1 Processor -- start 2024-03-29T11:29:44.709 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.717+0000 7febb8da3640 1 -- start start 2024-03-29T11:29:44.709 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.717+0000 7febb8da3640 10 monclient: init 2024-03-29T11:29:44.712 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.718+0000 7febb8da3640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:44.712 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.718+0000 7febb8da3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:29:44.712 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.718+0000 7febb8da3640 10 monclient(hunting): picked mon.noname-a con 0x7febb40b46b0 addr v1:172.21.15.104:6789/0 2024-03-29T11:29:44.712 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.718+0000 7febb8da3640 10 monclient(hunting): picked mon.noname-b con 0x7febb40aeae0 addr v1:172.21.15.104:6790/0 2024-03-29T11:29:44.712 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.718+0000 7febb8da3640 10 monclient(hunting): picked mon.noname-c con 0x7febb413fa10 addr v1:172.21.15.119:6789/0 2024-03-29T11:29:44.712 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.718+0000 7febb8da3640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb40b06a0 con 0x7febb40b46b0 2024-03-29T11:29:44.712 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.718+0000 7febb8da3640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb40b4230 con 0x7febb413fa10 2024-03-29T11:29:44.712 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.718+0000 7febb8da3640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb40b0910 con 0x7febb40aeae0 2024-03-29T11:29:44.712 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.718+0000 7febb8da3640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:44.712 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.718+0000 7febb8da3640 10 monclient(hunting): authenticate will time out at 1565.969542s 2024-03-29T11:29:44.715 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.722+0000 7febb259c640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7febb40b46b0 0x7febb40b3b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:51849/0 (socket says 172.21.15.104:51849) 2024-03-29T11:29:44.715 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.722+0000 7febb259c640 1 -- 172.21.15.104:0/1337647638 learned_addr learned my addr 172.21.15.104:0/1337647638 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:29:44.715 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.723+0000 7febb159a640 1 -- 172.21.15.104:0/1337647638 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4288982144 0 0) 0x7febb40b06a0 con 0x7febb40b46b0 2024-03-29T11:29:44.715 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.723+0000 7febb159a640 10 monclient(hunting): _negotiate 2024-03-29T11:29:44.715 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.723+0000 7febb159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:44.715 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.723+0000 7febb159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:44.715 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.723+0000 7febb159a640 10 monclient(hunting): my global_id is 4575 2024-03-29T11:29:44.715 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.723+0000 7febb159a640 1 -- 172.21.15.104:0/1337647638 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb9c003520 con 0x7febb40b46b0 2024-03-29T11:29:44.715 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.723+0000 7febb159a640 1 -- 172.21.15.104:0/1337647638 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 421873410 0 0) 0x7feb9c003520 con 0x7febb40b46b0 2024-03-29T11:29:44.716 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.723+0000 7febb159a640 1 -- 172.21.15.104:0/1337647638 >> v1:172.21.15.104:6790/0 conn(0x7febb40aeae0 legacy=0x7febb413f300 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:44.716 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.724+0000 7febb159a640 1 -- 172.21.15.104:0/1337647638 >> v1:172.21.15.119:6789/0 conn(0x7febb413fa10 legacy=0x7febb414d9e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:44.718 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.725+0000 7febb159a640 10 monclient: _finish_hunting 0 2024-03-29T11:29:44.718 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.725+0000 7febb159a640 1 monclient: found mon.noname-a 2024-03-29T11:29:44.718 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.725+0000 7febb159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:44.718 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.725+0000 7febb159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.104:6789/0 2024-03-29T11:29:44.718 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.725+0000 7febb159a640 1 -- 172.21.15.104:0/1337647638 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febb40af6d0 con 0x7febb40b46b0 2024-03-29T11:29:44.718 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.725+0000 7febb159a640 10 monclient: _finish_auth 0 2024-03-29T11:29:44.718 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.725+0000 7febb159a640 10 monclient: _check_auth_tickets 2024-03-29T11:29:44.718 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.725+0000 7febb159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:44.718 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.726+0000 7febb8da3640 5 monclient: authenticate success, global_id 4575 2024-03-29T11:29:44.718 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.726+0000 7febb8da3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:44.719 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.726+0000 7febb159a640 1 -- 172.21.15.104:0/1337647638 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7febac001a10 con 0x7febb40b46b0 2024-03-29T11:29:44.719 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.726+0000 7febb159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:44.719 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.726+0000 7febb159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:44.719 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.726+0000 7febb159a640 10 monclient: dump: 2024-03-29T11:29:44.719 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:29:44.719 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:29:44.719 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:44.719 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:44.719 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:44.719 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:44.719 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:29:44.719 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:29:44.719 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:29:44.720 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:44.720 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.726+0000 7febb8da3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:44.720 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.726+0000 7febb159a640 1 -- 172.21.15.104:0/1337647638 <== mon.0 v1:172.21.15.104:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7febac001ca0 con 0x7febb40b46b0 2024-03-29T11:29:44.720 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.726+0000 7febb159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:44.721 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.726+0000 7febb8da3640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:44.721 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.726+0000 7febb8da3640 4 set_mon_vals no callback set 2024-03-29T11:29:44.721 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.727+0000 7febb8da3640 10 monclient: shutdown 2024-03-29T11:29:44.721 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.727+0000 7febb8da3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:44.721 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.727+0000 7febb8da3640 1 -- 172.21.15.104:0/1337647638 >> v1:172.21.15.104:6789/0 conn(0x7febb40b46b0 legacy=0x7febb40b3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:44.721 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.728+0000 7febb8da3640 1 -- 172.21.15.104:0/1337647638 shutdown_connections 2024-03-29T11:29:44.721 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.728+0000 7febb8da3640 1 -- 172.21.15.104:0/1337647638 >> 172.21.15.104:0/1337647638 conn(0x7febb4066a00 msgr2=0x7febb4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:44.721 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.730+0000 7febb8da3640 1 -- 172.21.15.104:0/1337647638 shutdown_connections 2024-03-29T11:29:44.722 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.730+0000 7febb8da3640 1 -- 172.21.15.104:0/1337647638 wait complete. 2024-03-29T11:29:44.724 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.731+0000 7febb8da3640 10 monclient: build_initial_monmap 2024-03-29T11:29:44.724 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.731+0000 7febb8da3640 10 monclient: monmap: 2024-03-29T11:29:44.724 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:29:44.724 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:44.724 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:29:44.724 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:29:44.724 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:44.724 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:44.724 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:29:44.724 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:29:44.724 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:29:44.724 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:44.725 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.732+0000 7febb8da3640 1 librados: starting msgr at 2024-03-29T11:29:44.725 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.732+0000 7febb8da3640 1 librados: starting objecter 2024-03-29T11:29:44.725 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.732+0000 7febb8da3640 1 Processor -- start 2024-03-29T11:29:44.725 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.732+0000 7febb8da3640 1 -- start start 2024-03-29T11:29:44.725 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.732+0000 7febb8da3640 1 librados: setting wanted keys 2024-03-29T11:29:44.725 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.732+0000 7febb8da3640 1 librados: calling monclient init 2024-03-29T11:29:44.725 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.732+0000 7febb8da3640 10 monclient: init 2024-03-29T11:29:44.726 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.734+0000 7febb8da3640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:44.726 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.734+0000 7febb8da3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:29:44.726 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.734+0000 7febb8da3640 10 monclient(hunting): picked mon.noname-a con 0x7febb40aeae0 addr v1:172.21.15.104:6789/0 2024-03-29T11:29:44.726 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.734+0000 7febb8da3640 10 monclient(hunting): picked mon.noname-c con 0x7febb40b46b0 addr v1:172.21.15.104:6790/0 2024-03-29T11:29:44.726 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.734+0000 7febb8da3640 10 monclient(hunting): picked mon.noname-b con 0x7febb413fa10 addr v1:172.21.15.119:6789/0 2024-03-29T11:29:44.726 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.734+0000 7febb8da3640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb41464c0 con 0x7febb40aeae0 2024-03-29T11:29:44.726 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.734+0000 7febb8da3640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb4145a70 con 0x7febb413fa10 2024-03-29T11:29:44.726 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.734+0000 7febb8da3640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb4145430 con 0x7febb40b46b0 2024-03-29T11:29:44.726 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.734+0000 7febb8da3640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:44.727 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.734+0000 7febb8da3640 10 monclient(hunting): authenticate will time out at 1565.985636s 2024-03-29T11:29:44.734 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.741+0000 7febb259c640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7febb40aeae0 0x7febb41dab60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59547/0 (socket says 172.21.15.104:59547) 2024-03-29T11:29:44.734 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.741+0000 7febb259c640 1 -- 172.21.15.104:0/2063266307 learned_addr learned my addr 172.21.15.104:0/2063266307 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:29:44.734 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.741+0000 7feba2ffd640 10 client.?.objecter ms_handle_connect 0x7febb40aeae0 2024-03-29T11:29:44.734 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.741+0000 7feba2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:44.734 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.742+0000 7feba2ffd640 10 client.?.objecter ms_handle_connect 0x7febb413fa10 2024-03-29T11:29:44.734 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.742+0000 7feba2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:44.735 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.743+0000 7feba2ffd640 1 -- 172.21.15.104:0/2063266307 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3700864059 0 0) 0x7febb41464c0 con 0x7febb40aeae0 2024-03-29T11:29:44.735 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.743+0000 7feba2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:44.735 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.743+0000 7feba2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:44.735 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.743+0000 7feba2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:44.735 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.743+0000 7feba2ffd640 10 monclient(hunting): my global_id is 4578 2024-03-29T11:29:44.735 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.743+0000 7feba2ffd640 1 -- 172.21.15.104:0/2063266307 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb94004eb0 con 0x7febb40aeae0 2024-03-29T11:29:44.735 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.743+0000 7feba2ffd640 1 -- 172.21.15.104:0/2063266307 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2334329881 0 0) 0x7feb94004eb0 con 0x7febb40aeae0 2024-03-29T11:29:44.735 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.743+0000 7feba2ffd640 1 -- 172.21.15.104:0/2063266307 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb94005300 con 0x7febb40aeae0 2024-03-29T11:29:44.735 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.743+0000 7feba2ffd640 1 -- 172.21.15.104:0/2063266307 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7febac004ea0 con 0x7febb40aeae0 2024-03-29T11:29:44.736 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.743+0000 7feba2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:44.736 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.743+0000 7feba2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:44.736 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.743+0000 7feba2ffd640 10 monclient(hunting): dump: 2024-03-29T11:29:44.736 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:29:44.736 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:29:44.736 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:44.736 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:44.736 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:44.736 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:44.736 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:29:44.736 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:29:44.736 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:29:44.736 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:44.736 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.743+0000 7feba2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:44.739 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.744+0000 7feba2ffd640 10 client.?.objecter ms_handle_connect 0x7febb40b46b0 2024-03-29T11:29:44.739 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.744+0000 7feba2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:44.739 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.744+0000 7feba2ffd640 1 -- 172.21.15.104:0/2063266307 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3890945199 0 0) 0x7febb4145a70 con 0x7febb413fa10 2024-03-29T11:29:44.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.744+0000 7feba2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:44.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.744+0000 7feba2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:44.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.744+0000 7feba2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:44.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.744+0000 7feba2ffd640 10 monclient(hunting): my global_id is 4579 2024-03-29T11:29:44.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.744+0000 7feba2ffd640 1 -- 172.21.15.104:0/2063266307 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb94004eb0 con 0x7febb413fa10 2024-03-29T11:29:44.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.745+0000 7feba2ffd640 1 -- 172.21.15.104:0/2063266307 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2203963035 0 0) 0x7febb4145430 con 0x7febb40b46b0 2024-03-29T11:29:44.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.745+0000 7feba2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:44.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.745+0000 7feba2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:44.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.745+0000 7feba2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:44.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.745+0000 7feba2ffd640 10 monclient(hunting): my global_id is 4628 2024-03-29T11:29:44.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.745+0000 7feba2ffd640 1 -- 172.21.15.104:0/2063266307 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febb4145a70 con 0x7febb40b46b0 2024-03-29T11:29:44.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.746+0000 7feba2ffd640 1 -- 172.21.15.104:0/2063266307 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1485700075 0 0) 0x7feb94004eb0 con 0x7febb413fa10 2024-03-29T11:29:44.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.746+0000 7feba2ffd640 1 -- 172.21.15.104:0/2063266307 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb94005ca0 con 0x7febb413fa10 2024-03-29T11:29:44.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.746+0000 7feba2ffd640 1 -- 172.21.15.104:0/2063266307 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7feba8001730 con 0x7febb413fa10 2024-03-29T11:29:44.741 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.746+0000 7feba2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:44.741 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.746+0000 7feba2ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:44.741 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.746+0000 7feba2ffd640 10 monclient(hunting): dump: 2024-03-29T11:29:44.741 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:29:44.741 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:29:44.741 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:44.741 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:44.741 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:44.741 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:44.741 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:29:44.741 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:29:44.741 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:29:44.741 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:44.745 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.749+0000 7feba2ffd640 1 -- 172.21.15.104:0/2063266307 <== mon.0 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4115471523 0 0) 0x7feb94005300 con 0x7febb40aeae0 2024-03-29T11:29:44.745 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.749+0000 7feba2ffd640 1 -- 172.21.15.104:0/2063266307 >> v1:172.21.15.104:6790/0 conn(0x7febb40b46b0 legacy=0x7febb41ded70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:44.745 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.749+0000 7feba2ffd640 1 -- 172.21.15.104:0/2063266307 >> v1:172.21.15.119:6789/0 conn(0x7febb413fa10 legacy=0x7febb41df480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:44.745 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.749+0000 7feba2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:29:44.745 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.749+0000 7feba2ffd640 1 monclient: found mon.a 2024-03-29T11:29:44.745 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.749+0000 7feba2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:44.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.749+0000 7feba2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:29:44.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.749+0000 7feba2ffd640 1 -- 172.21.15.104:0/2063266307 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febb41455c0 con 0x7febb40aeae0 2024-03-29T11:29:44.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.749+0000 7feba2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:29:44.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.749+0000 7feba2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:29:44.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.749+0000 7feba2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:44.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.749+0000 7febb8da3640 5 monclient: authenticate success, global_id 4578 2024-03-29T11:29:44.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.749+0000 7febb8da3640 10 monclient: _renew_subs 2024-03-29T11:29:44.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.749+0000 7febb8da3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:29:44.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.749+0000 7febb8da3640 1 -- 172.21.15.104:0/2063266307 --> v1:172.21.15.104:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febb4144890 con 0x7febb40aeae0 2024-03-29T11:29:44.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.749+0000 7febb8da3640 10 client.4578.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:44.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.749+0000 7febb8da3640 10 monclient: _renew_subs 2024-03-29T11:29:44.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.749+0000 7febb8da3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:29:44.746 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.749+0000 7febb8da3640 1 -- 172.21.15.104:0/2063266307 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7febb4142050 con 0x7febb40aeae0 2024-03-29T11:29:44.747 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.749+0000 7febb8da3640 1 librados: init done 2024-03-29T11:29:44.747 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.750+0000 7febb8da3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:44.747 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.751+0000 7febb8da3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:44.747 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.751+0000 7febb8da3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:29:44.747 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.751+0000 7febb8da3640 1 -- 172.21.15.104:0/2063266307 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febb4075f10 con 0x7febb40aeae0 2024-03-29T11:29:44.747 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.752+0000 7feba2ffd640 1 -- 172.21.15.104:0/2063266307 <== mon.0 v1:172.21.15.104:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7febac001ed0 con 0x7febb40aeae0 2024-03-29T11:29:44.747 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.752+0000 7feba2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:44.747 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.752+0000 7feba2ffd640 1 -- 172.21.15.104:0/2063266307 <== mon.0 v1:172.21.15.104:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7febac005450 con 0x7febb40aeae0 2024-03-29T11:29:44.747 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.752+0000 7feba2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:44.747 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.752+0000 7feba2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:44.747 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.752+0000 7feba2ffd640 10 monclient: dump: 2024-03-29T11:29:44.747 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:29:44.747 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:29:44.747 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:44.747 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:44.748 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:44.748 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:44.748 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:29:44.748 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:29:44.748 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:29:44.748 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:44.748 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.753+0000 7feba2ffd640 1 -- 172.21.15.104:0/2063266307 <== mon.0 v1:172.21.15.104:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7febac005630 con 0x7febb40aeae0 2024-03-29T11:29:44.748 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.754+0000 7febb1d9b640 10 monclient: get_auth_request con 0x7feb9406e250 auth_method 0 2024-03-29T11:29:44.749 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.755+0000 7feba3fff640 4 set_mon_vals no callback set 2024-03-29T11:29:44.749 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.757+0000 7feba2ffd640 1 -- 172.21.15.104:0/2063266307 <== mon.0 v1:172.21.15.104:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 668040924 0 0) 0x7febac054d00 con 0x7febb40aeae0 2024-03-29T11:29:44.749 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.757+0000 7feba2ffd640 10 client.4578.objecter ms_dispatch 0x7febb4000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:29:44.750 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.757+0000 7feba2ffd640 3 client.4578.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:29:44.750 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.757+0000 7feba2ffd640 3 client.4578.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:29:44.750 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.757+0000 7feba2ffd640 20 client.4578.objecter dump_active .. 0 homeless 2024-03-29T11:29:44.750 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.758+0000 7feba2ffd640 10 client.4578.objecter ms_handle_connect 0x7feb9406e250 2024-03-29T11:29:44.753 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.761+0000 7feba2ffd640 1 -- 172.21.15.104:0/2063266307 <== mon.0 v1:172.21.15.104:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7febac050fa0 con 0x7febb40aeae0 2024-03-29T11:29:44.753 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.761+0000 7feba2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:44.753 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:44.761+0000 7feba2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:45.075 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:45.083+0000 7febb8da3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:45.075 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:45.084+0000 7febb8da3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:45.075 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:45.084+0000 7febb8da3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:29:45.075 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:45.084+0000 7febb8da3640 1 -- 172.21.15.104:0/2063266307 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7febb40afa70 con 0x7febb40aeae0 2024-03-29T11:29:45.077 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:45.086+0000 7feba2ffd640 1 -- 172.21.15.104:0/2063266307 <== mon.0 v1:172.21.15.104:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3673681244) 0x7febac051140 con 0x7febb40aeae0 2024-03-29T11:29:45.077 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:45.086+0000 7feba2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:45.077 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:45.086+0000 7feba2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:45.079 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:29:45.079 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":11,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:29:29.328850+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:29.328850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:45.083 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:45.090+0000 7febb8da3640 10 librados: watch_flush enter 2024-03-29T11:29:45.083 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:45.090+0000 7febb8da3640 10 librados: watch_flush exit 2024-03-29T11:29:45.084 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:45.090+0000 7febb8da3640 20 client.4578.objecter shutdown clearing up homeless session... 2024-03-29T11:29:45.084 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:45.090+0000 7febb8da3640 10 client.4578.objecter successfully canceled tick 2024-03-29T11:29:45.084 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:45.090+0000 7febb8da3640 1 -- 172.21.15.104:0/2063266307 >> v1:172.21.15.104:6816/3341126660 conn(0x7feb9406e250 legacy=0x7feb94070710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:45.084 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:45.090+0000 7febb8da3640 10 monclient: shutdown 2024-03-29T11:29:45.084 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:45.090+0000 7febb8da3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:45.084 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:45.090+0000 7febb8da3640 1 -- 172.21.15.104:0/2063266307 >> v1:172.21.15.104:6789/0 conn(0x7febb40aeae0 legacy=0x7febb41dab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:45.085 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:45.092+0000 7febb8da3640 1 -- 172.21.15.104:0/2063266307 shutdown_connections 2024-03-29T11:29:45.085 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:45.092+0000 7febb8da3640 1 -- 172.21.15.104:0/2063266307 >> 172.21.15.104:0/2063266307 conn(0x7febb4066a00 msgr2=0x7febb413f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:45.085 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:45.093+0000 7febb8da3640 1 -- 172.21.15.104:0/2063266307 shutdown_connections 2024-03-29T11:29:45.085 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:45.093+0000 7febb8da3640 1 -- 172.21.15.104:0/2063266307 wait complete. 2024-03-29T11:29:45.085 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:45.093+0000 7febb8da3640 1 librados: shutdown 2024-03-29T11:29:45.103 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:51.911 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:52.059 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.067+0000 7f0ed759e640 10 monclient: get_monmap_and_config 2024-03-29T11:29:52.059 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.067+0000 7f0ed759e640 10 monclient: build_initial_monmap 2024-03-29T11:29:52.059 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.067+0000 7f0ed759e640 10 monclient: monmap: 2024-03-29T11:29:52.059 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:29:52.059 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:29:52.059 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:29:52.068468+0000 2024-03-29T11:29:52.060 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:29:52.068468+0000 2024-03-29T11:29:52.060 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:52.060 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:52.060 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:29:52.060 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:29:52.060 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:29:52.060 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:52.062 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.070+0000 7f0ed759e640 1 Processor -- start 2024-03-29T11:29:52.062 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.070+0000 7f0ed759e640 1 -- start start 2024-03-29T11:29:52.062 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.070+0000 7f0ed759e640 10 monclient: init 2024-03-29T11:29:52.063 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.071+0000 7f0ed759e640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:52.063 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.071+0000 7f0ed759e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:29:52.063 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.071+0000 7f0ed759e640 10 monclient(hunting): picked mon.noname-c con 0x7f0ed002b4c0 addr v1:172.21.15.119:6789/0 2024-03-29T11:29:52.063 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.071+0000 7f0ed759e640 10 monclient(hunting): picked mon.noname-b con 0x7f0ed000b8a0 addr v1:172.21.15.104:6790/0 2024-03-29T11:29:52.063 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.071+0000 7f0ed759e640 10 monclient(hunting): picked mon.noname-a con 0x7f0ed002a5a0 addr v1:172.21.15.104:6789/0 2024-03-29T11:29:52.063 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.071+0000 7f0ed759e640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ed00c11a0 con 0x7f0ed002a5a0 2024-03-29T11:29:52.063 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.071+0000 7f0ed759e640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ed002c830 con 0x7f0ed002b4c0 2024-03-29T11:29:52.063 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.071+0000 7f0ed759e640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ed002afa0 con 0x7f0ed000b8a0 2024-03-29T11:29:52.064 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.071+0000 7f0ed759e640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:52.064 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.071+0000 7f0ed759e640 10 monclient(hunting): authenticate will time out at 1573.322849s 2024-03-29T11:29:52.065 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.072+0000 7f0ed5d9b640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f0ed000b8a0 0x7f0ed00c0a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:39688/0 (socket says 172.21.15.104:39688) 2024-03-29T11:29:52.065 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.072+0000 7f0ed5d9b640 1 -- 172.21.15.104:0/1828933914 learned_addr learned my addr 172.21.15.104:0/1828933914 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:29:52.065 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.073+0000 7f0ed559a640 1 -- 172.21.15.104:0/1828933914 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1664568878 0 0) 0x7f0ed002afa0 con 0x7f0ed000b8a0 2024-03-29T11:29:52.065 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.073+0000 7f0ed559a640 10 monclient(hunting): _negotiate 2024-03-29T11:29:52.065 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.073+0000 7f0ed559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:52.065 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.073+0000 7f0ed559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:52.065 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.073+0000 7f0ed559a640 10 monclient(hunting): my global_id is 4631 2024-03-29T11:29:52.065 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.073+0000 7f0ed559a640 1 -- 172.21.15.104:0/1828933914 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ec0005730 con 0x7f0ed000b8a0 2024-03-29T11:29:52.066 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.073+0000 7f0ed559a640 1 -- 172.21.15.104:0/1828933914 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3090871135 0 0) 0x7f0ec0005730 con 0x7f0ed000b8a0 2024-03-29T11:29:52.066 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.073+0000 7f0ed559a640 1 -- 172.21.15.104:0/1828933914 >> v1:172.21.15.119:6789/0 conn(0x7f0ed002b4c0 legacy=0x7f0ed0006890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:52.067 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.075+0000 7f0ed559a640 1 -- 172.21.15.104:0/1828933914 >> v1:172.21.15.104:6789/0 conn(0x7f0ed002a5a0 legacy=0x7f0ed00c6d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:52.070 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.076+0000 7f0ed559a640 10 monclient: _finish_hunting 0 2024-03-29T11:29:52.070 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.076+0000 7f0ed559a640 1 monclient: found mon.noname-b 2024-03-29T11:29:52.070 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.076+0000 7f0ed559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:52.070 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.076+0000 7f0ed559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.104:6790/0 2024-03-29T11:29:52.070 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.076+0000 7f0ed559a640 1 -- 172.21.15.104:0/1828933914 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ed002aba0 con 0x7f0ed000b8a0 2024-03-29T11:29:52.070 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.076+0000 7f0ed559a640 10 monclient: _finish_auth 0 2024-03-29T11:29:52.070 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.076+0000 7f0ed559a640 10 monclient: _check_auth_tickets 2024-03-29T11:29:52.070 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.076+0000 7f0ed559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:52.070 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.076+0000 7f0ed759e640 5 monclient: authenticate success, global_id 4631 2024-03-29T11:29:52.070 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.076+0000 7f0ed759e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:52.070 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.076+0000 7f0ed559a640 1 -- 172.21.15.104:0/1828933914 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f0ecc001690 con 0x7f0ed000b8a0 2024-03-29T11:29:52.071 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.076+0000 7f0ed559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:52.071 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.076+0000 7f0ed559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:52.071 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.076+0000 7f0ed559a640 10 monclient: dump: 2024-03-29T11:29:52.071 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:29:52.071 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:29:52.071 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:52.071 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:52.071 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:52.071 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:52.071 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:29:52.071 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:29:52.071 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:29:52.071 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:52.071 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.076+0000 7f0ed759e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:52.071 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.076+0000 7f0ed559a640 1 -- 172.21.15.104:0/1828933914 <== mon.2 v1:172.21.15.104:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ecc001920 con 0x7f0ed000b8a0 2024-03-29T11:29:52.072 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.076+0000 7f0ed559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:52.072 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.076+0000 7f0ed759e640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:52.072 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.076+0000 7f0ed759e640 4 set_mon_vals no callback set 2024-03-29T11:29:52.072 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.076+0000 7f0ed559a640 1 -- 172.21.15.104:0/1828933914 <== mon.2 v1:172.21.15.104:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f0ecc001cb0 con 0x7f0ed000b8a0 2024-03-29T11:29:52.072 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.076+0000 7f0ed559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:52.072 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.076+0000 7f0ed559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:52.072 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.076+0000 7f0ed559a640 10 monclient: dump: 2024-03-29T11:29:52.072 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:29:52.072 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:29:52.072 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:52.072 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:52.072 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:52.072 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:52.072 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:29:52.072 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:29:52.073 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:29:52.073 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:52.073 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.077+0000 7f0ed759e640 10 monclient: shutdown 2024-03-29T11:29:52.073 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.077+0000 7f0ed759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:52.073 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.077+0000 7f0ed759e640 1 -- 172.21.15.104:0/1828933914 >> v1:172.21.15.104:6790/0 conn(0x7f0ed000b8a0 legacy=0x7f0ed00c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:52.073 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.078+0000 7f0ed759e640 1 -- 172.21.15.104:0/1828933914 shutdown_connections 2024-03-29T11:29:52.073 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.078+0000 7f0ed759e640 1 -- 172.21.15.104:0/1828933914 >> 172.21.15.104:0/1828933914 conn(0x7f0ed00058f0 msgr2=0x7f0ed0005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:52.073 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.078+0000 7f0ed759e640 1 -- 172.21.15.104:0/1828933914 shutdown_connections 2024-03-29T11:29:52.073 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.078+0000 7f0ed759e640 1 -- 172.21.15.104:0/1828933914 wait complete. 2024-03-29T11:29:52.076 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.079+0000 7f0ed759e640 10 monclient: build_initial_monmap 2024-03-29T11:29:52.076 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.079+0000 7f0ed759e640 10 monclient: monmap: 2024-03-29T11:29:52.076 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:29:52.076 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:52.076 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:29:52.076 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:29:52.076 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:52.077 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:52.077 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:29:52.077 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:29:52.077 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:29:52.077 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:52.077 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.079+0000 7f0ed759e640 1 librados: starting msgr at 2024-03-29T11:29:52.077 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.079+0000 7f0ed759e640 1 librados: starting objecter 2024-03-29T11:29:52.077 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.080+0000 7f0ed759e640 1 Processor -- start 2024-03-29T11:29:52.077 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.080+0000 7f0ed759e640 1 -- start start 2024-03-29T11:29:52.077 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.080+0000 7f0ed759e640 1 librados: setting wanted keys 2024-03-29T11:29:52.077 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.080+0000 7f0ed759e640 1 librados: calling monclient init 2024-03-29T11:29:52.077 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.080+0000 7f0ed759e640 10 monclient: init 2024-03-29T11:29:52.077 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.081+0000 7f0ed759e640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:52.078 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.081+0000 7f0ed759e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:29:52.078 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.081+0000 7f0ed759e640 10 monclient(hunting): picked mon.noname-c con 0x7f0ed000b8a0 addr v1:172.21.15.104:6790/0 2024-03-29T11:29:52.078 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.081+0000 7f0ed759e640 10 monclient(hunting): picked mon.noname-a con 0x7f0ed002a5a0 addr v1:172.21.15.104:6789/0 2024-03-29T11:29:52.078 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.081+0000 7f0ed659c640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f0ed000b8a0 0x7f0ed01643f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:41060/0 (socket says 172.21.15.104:41060) 2024-03-29T11:29:52.078 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.081+0000 7f0ed659c640 1 -- 172.21.15.104:0/1681305067 learned_addr learned my addr 172.21.15.104:0/1681305067 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:29:52.078 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.081+0000 7f0ec6ffd640 10 client.?.objecter ms_handle_connect 0x7f0ed000b8a0 2024-03-29T11:29:52.078 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.081+0000 7f0ec6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:52.078 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.082+0000 7f0ed759e640 10 monclient(hunting): picked mon.noname-b con 0x7f0ed002b4c0 addr v1:172.21.15.119:6789/0 2024-03-29T11:29:52.078 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.082+0000 7f0ed759e640 1 -- 172.21.15.104:0/1681305067 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ed002eac0 con 0x7f0ed002a5a0 2024-03-29T11:29:52.078 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.082+0000 7f0ed759e640 1 -- 172.21.15.104:0/1681305067 --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ed002eda0 con 0x7f0ed002b4c0 2024-03-29T11:29:52.078 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.082+0000 7f0ed759e640 1 -- 172.21.15.104:0/1681305067 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ed00bf290 con 0x7f0ed000b8a0 2024-03-29T11:29:52.078 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.082+0000 7f0ed759e640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:52.078 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.082+0000 7f0ed759e640 10 monclient(hunting): authenticate will time out at 1573.333600s 2024-03-29T11:29:52.078 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.083+0000 7f0ec6ffd640 1 -- 172.21.15.104:0/1681305067 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3308403833 0 0) 0x7f0ed00bf290 con 0x7f0ed000b8a0 2024-03-29T11:29:52.078 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.083+0000 7f0ec6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:52.079 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.083+0000 7f0ec6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:52.079 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.083+0000 7f0ec6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:52.079 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.083+0000 7f0ec6ffd640 10 monclient(hunting): my global_id is 4634 2024-03-29T11:29:52.079 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.083+0000 7f0ec6ffd640 1 -- 172.21.15.104:0/1681305067 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0eb8004eb0 con 0x7f0ed000b8a0 2024-03-29T11:29:52.080 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.086+0000 7f0ec6ffd640 1 -- 172.21.15.104:0/1681305067 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3012317580 0 0) 0x7f0eb8004eb0 con 0x7f0ed000b8a0 2024-03-29T11:29:52.080 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.086+0000 7f0ec6ffd640 1 -- 172.21.15.104:0/1681305067 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ed00bf290 con 0x7f0ed000b8a0 2024-03-29T11:29:52.080 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.086+0000 7f0ec6ffd640 1 -- 172.21.15.104:0/1681305067 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f0ec8001870 con 0x7f0ed000b8a0 2024-03-29T11:29:52.080 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.086+0000 7f0ec6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:52.080 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.086+0000 7f0ec6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:52.080 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.086+0000 7f0ec6ffd640 10 monclient(hunting): dump: 2024-03-29T11:29:52.081 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:29:52.081 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:29:52.081 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:52.081 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:52.081 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:52.081 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:52.081 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:29:52.081 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:29:52.081 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:29:52.081 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:52.081 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.086+0000 7f0ec6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:52.081 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.086+0000 7f0ec6ffd640 10 client.?.objecter ms_handle_connect 0x7f0ed002b4c0 2024-03-29T11:29:52.081 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.086+0000 7f0ec6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:52.081 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.086+0000 7f0ec6ffd640 1 -- 172.21.15.104:0/1681305067 <== mon.2 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 176308526 0 0) 0x7f0ed00bf290 con 0x7f0ed000b8a0 2024-03-29T11:29:52.081 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.087+0000 7f0ec6ffd640 1 -- 172.21.15.104:0/1681305067 >> v1:172.21.15.119:6789/0 conn(0x7f0ed002b4c0 legacy=0x7f0ed016b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:52.082 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.087+0000 7f0ec6ffd640 1 -- 172.21.15.104:0/1681305067 >> v1:172.21.15.104:6789/0 conn(0x7f0ed002a5a0 legacy=0x7f0ed00c1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:52.082 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.087+0000 7f0ec6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:29:52.082 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.087+0000 7f0ec6ffd640 1 monclient: found mon.c 2024-03-29T11:29:52.082 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.087+0000 7f0ec6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:52.082 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.087+0000 7f0ec6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:29:52.082 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.087+0000 7f0ec6ffd640 1 -- 172.21.15.104:0/1681305067 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ed00c2210 con 0x7f0ed000b8a0 2024-03-29T11:29:52.082 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.087+0000 7f0ec6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:29:52.082 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.087+0000 7f0ec6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:29:52.082 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.087+0000 7f0ec6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:52.082 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.087+0000 7f0ec6ffd640 10 client.?.objecter ms_handle_connect 0x7f0ed002a5a0 2024-03-29T11:29:52.082 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.087+0000 7f0ec6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:52.082 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.087+0000 7f0ed759e640 5 monclient: authenticate success, global_id 4634 2024-03-29T11:29:52.082 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.087+0000 7f0ed759e640 10 monclient: _renew_subs 2024-03-29T11:29:52.082 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.087+0000 7f0ed759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:29:52.083 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.087+0000 7f0ed759e640 1 -- 172.21.15.104:0/1681305067 --> v1:172.21.15.104:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ed00c4350 con 0x7f0ed000b8a0 2024-03-29T11:29:52.083 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.087+0000 7f0ed759e640 10 client.4634.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:52.083 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.087+0000 7f0ed759e640 10 monclient: _renew_subs 2024-03-29T11:29:52.083 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.087+0000 7f0ed759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:29:52.083 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.087+0000 7f0ed759e640 1 -- 172.21.15.104:0/1681305067 --> v1:172.21.15.104:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ed00c4640 con 0x7f0ed000b8a0 2024-03-29T11:29:52.083 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.087+0000 7f0ed759e640 1 librados: init done 2024-03-29T11:29:52.085 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.089+0000 7f0ed759e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:52.085 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.089+0000 7f0ed759e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:52.085 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.089+0000 7f0ed759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:29:52.085 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.089+0000 7f0ed759e640 1 -- 172.21.15.104:0/1681305067 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ed00c49b0 con 0x7f0ed000b8a0 2024-03-29T11:29:52.085 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.089+0000 7f0ec6ffd640 1 -- 172.21.15.104:0/1681305067 <== mon.2 v1:172.21.15.104:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ec80019d0 con 0x7f0ed000b8a0 2024-03-29T11:29:52.086 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.089+0000 7f0ec6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:52.086 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.089+0000 7f0ec6ffd640 1 -- 172.21.15.104:0/1681305067 <== mon.2 v1:172.21.15.104:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f0ec8004ae0 con 0x7f0ed000b8a0 2024-03-29T11:29:52.086 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.089+0000 7f0ec6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:52.086 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.089+0000 7f0ec6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:52.086 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.089+0000 7f0ec6ffd640 10 monclient: dump: 2024-03-29T11:29:52.086 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:29:52.086 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:29:52.086 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:52.086 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:52.086 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:52.086 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:52.086 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:29:52.086 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:29:52.086 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:29:52.087 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:52.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.089+0000 7f0ed4d99640 4 set_mon_vals no callback set 2024-03-29T11:29:52.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.091+0000 7f0ec6ffd640 1 -- 172.21.15.104:0/1681305067 <== mon.2 v1:172.21.15.104:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f0ec8004c40 con 0x7f0ed000b8a0 2024-03-29T11:29:52.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.092+0000 7f0ec6ffd640 1 -- 172.21.15.104:0/1681305067 <== mon.2 v1:172.21.15.104:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 668040924 0 0) 0x7f0ec8084ab0 con 0x7f0ed000b8a0 2024-03-29T11:29:52.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.092+0000 7f0ec6ffd640 10 client.4634.objecter ms_dispatch 0x7f0ed8000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:29:52.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.092+0000 7f0ec6ffd640 3 client.4634.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:29:52.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.092+0000 7f0ec6ffd640 3 client.4634.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:29:52.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.092+0000 7f0ec6ffd640 20 client.4634.objecter dump_active .. 0 homeless 2024-03-29T11:29:52.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.092+0000 7f0ed5d9b640 10 monclient: get_auth_request con 0x7f0eb806db20 auth_method 0 2024-03-29T11:29:52.087 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.093+0000 7f0ec6ffd640 10 client.4634.objecter ms_handle_connect 0x7f0eb806db20 2024-03-29T11:29:52.092 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.100+0000 7f0ec6ffd640 1 -- 172.21.15.104:0/1681305067 <== mon.2 v1:172.21.15.104:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0ec80501c0 con 0x7f0ed000b8a0 2024-03-29T11:29:52.092 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.100+0000 7f0ec6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:52.092 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.100+0000 7f0ec6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:52.423 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.427+0000 7f0ed759e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:52.423 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.427+0000 7f0ed759e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:52.423 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.427+0000 7f0ed759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:29:52.423 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.427+0000 7f0ed759e640 1 -- 172.21.15.104:0/1681305067 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ed0004cd0 con 0x7f0ed000b8a0 2024-03-29T11:29:52.423 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.428+0000 7f0ec6ffd640 1 -- 172.21.15.104:0/1681305067 <== mon.2 v1:172.21.15.104:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3673681244) 0x7f0ec80522c0 con 0x7f0ed000b8a0 2024-03-29T11:29:52.423 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.428+0000 7f0ec6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:52.423 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.428+0000 7f0ec6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:52.424 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:29:52.424 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":11,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:29:29.328850+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:29.328850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:52.424 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.432+0000 7f0ed759e640 10 librados: watch_flush enter 2024-03-29T11:29:52.425 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.432+0000 7f0ed759e640 10 librados: watch_flush exit 2024-03-29T11:29:52.425 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.432+0000 7f0ed759e640 20 client.4634.objecter shutdown clearing up homeless session... 2024-03-29T11:29:52.425 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.432+0000 7f0ed759e640 10 client.4634.objecter successfully canceled tick 2024-03-29T11:29:52.425 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.433+0000 7f0ed759e640 1 -- 172.21.15.104:0/1681305067 >> v1:172.21.15.104:6816/3341126660 conn(0x7f0eb806db20 legacy=0x7f0eb806ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:52.425 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.433+0000 7f0ed759e640 10 monclient: shutdown 2024-03-29T11:29:52.425 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.433+0000 7f0ed759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:52.425 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.433+0000 7f0ed759e640 1 -- 172.21.15.104:0/1681305067 >> v1:172.21.15.104:6790/0 conn(0x7f0ed000b8a0 legacy=0x7f0ed01643f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:52.425 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.433+0000 7f0ed759e640 1 -- 172.21.15.104:0/1681305067 shutdown_connections 2024-03-29T11:29:52.425 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.433+0000 7f0ed759e640 1 -- 172.21.15.104:0/1681305067 >> 172.21.15.104:0/1681305067 conn(0x7f0ed00058f0 msgr2=0x7f0ed0005cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:52.425 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.433+0000 7f0ed759e640 1 -- 172.21.15.104:0/1681305067 shutdown_connections 2024-03-29T11:29:52.425 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.433+0000 7f0ed759e640 1 -- 172.21.15.104:0/1681305067 wait complete. 2024-03-29T11:29:52.425 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:52.433+0000 7f0ed759e640 1 librados: shutdown 2024-03-29T11:29:52.442 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:59.250 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:59.400 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.408+0000 7f95fcd41640 10 monclient: get_monmap_and_config 2024-03-29T11:29:59.400 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.408+0000 7f95fcd41640 10 monclient: build_initial_monmap 2024-03-29T11:29:59.400 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.408+0000 7f95fcd41640 10 monclient: monmap: 2024-03-29T11:29:59.400 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:29:59.400 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:29:59.400 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:29:59.409253+0000 2024-03-29T11:29:59.401 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:29:59.409253+0000 2024-03-29T11:29:59.401 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:59.401 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:59.401 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:29:59.401 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:29:59.401 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:29:59.401 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:59.401 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.409+0000 7f95fcd41640 1 Processor -- start 2024-03-29T11:29:59.401 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.409+0000 7f95fcd41640 1 -- start start 2024-03-29T11:29:59.402 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.409+0000 7f95fcd41640 10 monclient: init 2024-03-29T11:29:59.402 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.410+0000 7f95fcd41640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:59.402 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.410+0000 7f95fcd41640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:29:59.402 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.410+0000 7f95fcd41640 10 monclient(hunting): picked mon.noname-a con 0x7f95f00066e0 addr v1:172.21.15.104:6789/0 2024-03-29T11:29:59.402 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.410+0000 7f95fcd41640 10 monclient(hunting): picked mon.noname-c con 0x7f95f00cf820 addr v1:172.21.15.119:6789/0 2024-03-29T11:29:59.402 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.410+0000 7f95fcd41640 10 monclient(hunting): picked mon.noname-b con 0x7f95f00c92b0 addr v1:172.21.15.104:6790/0 2024-03-29T11:29:59.402 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.410+0000 7f95fcd41640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95f002c100 con 0x7f95f00066e0 2024-03-29T11:29:59.402 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.410+0000 7f95fcd41640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95f002c3e0 con 0x7f95f00cf820 2024-03-29T11:29:59.403 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.410+0000 7f95fcd41640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95f002a030 con 0x7f95f00c92b0 2024-03-29T11:29:59.403 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.410+0000 7f95fcd41640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:59.403 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.410+0000 7f95fcd41640 10 monclient(hunting): authenticate will time out at 1580.661494s 2024-03-29T11:29:59.403 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.411+0000 7f95f7fff640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f95f00c92b0 0x7f95f00d74b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:42858/0 (socket says 172.21.15.104:42858) 2024-03-29T11:29:59.403 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.411+0000 7f95f7fff640 1 -- 172.21.15.104:0/3373313827 learned_addr learned my addr 172.21.15.104:0/3373313827 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:29:59.404 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.413+0000 7f95f67fc640 1 -- 172.21.15.104:0/3373313827 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2009329972 0 0) 0x7f95f002a030 con 0x7f95f00c92b0 2024-03-29T11:29:59.411 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.413+0000 7f95f67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:29:59.413 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.413+0000 7f95f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:59.413 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.413+0000 7f95f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:59.413 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.413+0000 7f95f67fc640 10 monclient(hunting): my global_id is 4637 2024-03-29T11:29:59.413 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.413+0000 7f95f67fc640 1 -- 172.21.15.104:0/3373313827 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95e8003520 con 0x7f95f00c92b0 2024-03-29T11:29:59.413 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.414+0000 7f95f67fc640 1 -- 172.21.15.104:0/3373313827 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3295535470 0 0) 0x7f95e8003520 con 0x7f95f00c92b0 2024-03-29T11:29:59.413 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.414+0000 7f95f67fc640 1 -- 172.21.15.104:0/3373313827 >> v1:172.21.15.119:6789/0 conn(0x7f95f00cf820 legacy=0x7f95f00c8ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:59.413 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.414+0000 7f95f67fc640 1 -- 172.21.15.104:0/3373313827 >> v1:172.21.15.104:6789/0 conn(0x7f95f00066e0 legacy=0x7f95f00cf110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:59.413 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.414+0000 7f95f67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:29:59.413 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.414+0000 7f95f67fc640 1 monclient: found mon.noname-b 2024-03-29T11:29:59.413 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.414+0000 7f95f67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:59.414 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.414+0000 7f95f67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.104:6790/0 2024-03-29T11:29:59.414 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.414+0000 7f95f67fc640 1 -- 172.21.15.104:0/3373313827 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95f0029ce0 con 0x7f95f00c92b0 2024-03-29T11:29:59.414 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.414+0000 7f95f67fc640 10 monclient: _finish_auth 0 2024-03-29T11:29:59.414 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.414+0000 7f95f67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:29:59.414 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.414+0000 7f95f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:59.414 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.414+0000 7f95fcd41640 5 monclient: authenticate success, global_id 4637 2024-03-29T11:29:59.414 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.414+0000 7f95fcd41640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:59.414 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.414+0000 7f95f67fc640 1 -- 172.21.15.104:0/3373313827 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f95ec001690 con 0x7f95f00c92b0 2024-03-29T11:29:59.414 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.414+0000 7f95f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:59.415 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.414+0000 7f95f67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:59.415 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.414+0000 7f95f67fc640 10 monclient: dump: 2024-03-29T11:29:59.415 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:29:59.415 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:29:59.415 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:59.415 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:59.415 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:59.415 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:59.415 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:29:59.415 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:29:59.416 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:29:59.416 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:59.416 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.414+0000 7f95fcd41640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:59.416 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.414+0000 7f95f67fc640 1 -- 172.21.15.104:0/3373313827 <== mon.2 v1:172.21.15.104:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f95ec001860 con 0x7f95f00c92b0 2024-03-29T11:29:59.416 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.414+0000 7f95f67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:59.416 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.414+0000 7f95fcd41640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:59.416 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.414+0000 7f95fcd41640 4 set_mon_vals no callback set 2024-03-29T11:29:59.416 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.414+0000 7f95f67fc640 1 -- 172.21.15.104:0/3373313827 <== mon.2 v1:172.21.15.104:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f95ec001c70 con 0x7f95f00c92b0 2024-03-29T11:29:59.416 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.414+0000 7f95f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:59.416 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.414+0000 7f95f67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:59.416 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.414+0000 7f95f67fc640 10 monclient: dump: 2024-03-29T11:29:59.416 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:29:59.417 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:29:59.417 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:59.417 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:59.417 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:59.417 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:59.417 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:29:59.417 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:29:59.417 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:29:59.417 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:59.417 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.415+0000 7f95fcd41640 10 monclient: shutdown 2024-03-29T11:29:59.417 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.415+0000 7f95fcd41640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:59.417 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.415+0000 7f95fcd41640 1 -- 172.21.15.104:0/3373313827 >> v1:172.21.15.104:6790/0 conn(0x7f95f00c92b0 legacy=0x7f95f00d74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:59.417 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.415+0000 7f95fcd41640 1 -- 172.21.15.104:0/3373313827 shutdown_connections 2024-03-29T11:29:59.417 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.415+0000 7f95fcd41640 1 -- 172.21.15.104:0/3373313827 >> 172.21.15.104:0/3373313827 conn(0x7f95f00058f0 msgr2=0x7f95f0005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:59.417 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.416+0000 7f95fcd41640 1 -- 172.21.15.104:0/3373313827 shutdown_connections 2024-03-29T11:29:59.418 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.416+0000 7f95fcd41640 1 -- 172.21.15.104:0/3373313827 wait complete. 2024-03-29T11:29:59.418 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.416+0000 7f95fcd41640 10 monclient: build_initial_monmap 2024-03-29T11:29:59.418 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.416+0000 7f95fcd41640 10 monclient: monmap: 2024-03-29T11:29:59.418 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:29:59.418 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:59.418 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:29:59.418 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:29:59.418 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:59.418 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:59.418 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:29:59.418 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:29:59.418 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:29:59.418 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:59.418 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.417+0000 7f95fcd41640 1 librados: starting msgr at 2024-03-29T11:29:59.418 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.417+0000 7f95fcd41640 1 librados: starting objecter 2024-03-29T11:29:59.419 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.417+0000 7f95fcd41640 1 Processor -- start 2024-03-29T11:29:59.419 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.417+0000 7f95fcd41640 1 -- start start 2024-03-29T11:29:59.419 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.417+0000 7f95fcd41640 1 librados: setting wanted keys 2024-03-29T11:29:59.419 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.417+0000 7f95fcd41640 1 librados: calling monclient init 2024-03-29T11:29:59.419 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.417+0000 7f95fcd41640 10 monclient: init 2024-03-29T11:29:59.421 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.418+0000 7f95fcd41640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:59.421 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.418+0000 7f95fcd41640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:29:59.421 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.418+0000 7f95fcd41640 10 monclient(hunting): picked mon.noname-b con 0x7f95f00066e0 addr v1:172.21.15.119:6789/0 2024-03-29T11:29:59.421 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.418+0000 7f95fcd41640 10 monclient(hunting): picked mon.noname-a con 0x7f95f00c92b0 addr v1:172.21.15.104:6789/0 2024-03-29T11:29:59.421 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.418+0000 7f95fcd41640 10 monclient(hunting): picked mon.noname-c con 0x7f95f00cf820 addr v1:172.21.15.104:6790/0 2024-03-29T11:29:59.422 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.418+0000 7f95fcd41640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95f00cbf20 con 0x7f95f00c92b0 2024-03-29T11:29:59.422 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.418+0000 7f95fcd41640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95f00c12c0 con 0x7f95f00066e0 2024-03-29T11:29:59.422 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.418+0000 7f95fcd41640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95f002d3d0 con 0x7f95f00cf820 2024-03-29T11:29:59.422 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.418+0000 7f95fcd41640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:59.422 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.418+0000 7f95fcd41640 10 monclient(hunting): authenticate will time out at 1580.669712s 2024-03-29T11:29:59.422 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.422+0000 7f95f7fff640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f95f00cf820 0x7f95f00c0bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:43606/0 (socket says 172.21.15.104:43606) 2024-03-29T11:29:59.422 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.422+0000 7f95f7fff640 1 -- 172.21.15.104:0/2250630968 learned_addr learned my addr 172.21.15.104:0/2250630968 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:29:59.422 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.422+0000 7f95f6ffd640 1 --1- 172.21.15.104:0/2250630968 >> v1:172.21.15.104:6789/0 conn(0x7f95f00c92b0 0x7f95f00c04a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:55525/0 (socket says 172.21.15.104:55525) 2024-03-29T11:29:59.422 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.423+0000 7f95e3fff640 10 client.?.objecter ms_handle_connect 0x7f95f00cf820 2024-03-29T11:29:59.422 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.423+0000 7f95e3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:59.422 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.423+0000 7f95e3fff640 10 client.?.objecter ms_handle_connect 0x7f95f00c92b0 2024-03-29T11:29:59.422 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.423+0000 7f95e3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:59.422 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.423+0000 7f95e3fff640 1 -- 172.21.15.104:0/2250630968 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3032885573 0 0) 0x7f95f00cbf20 con 0x7f95f00c92b0 2024-03-29T11:29:59.423 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.423+0000 7f95e3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:29:59.423 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.423+0000 7f95e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:59.423 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.423+0000 7f95e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:59.423 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.423+0000 7f95e3fff640 10 monclient(hunting): my global_id is 4581 2024-03-29T11:29:59.423 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.423+0000 7f95e3fff640 1 -- 172.21.15.104:0/2250630968 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95dc004eb0 con 0x7f95f00c92b0 2024-03-29T11:29:59.423 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.423+0000 7f95e3fff640 1 -- 172.21.15.104:0/2250630968 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2995566777 0 0) 0x7f95f002d3d0 con 0x7f95f00cf820 2024-03-29T11:29:59.423 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.423+0000 7f95e3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:29:59.423 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.423+0000 7f95e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:59.423 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.423+0000 7f95e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:59.423 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.423+0000 7f95e3fff640 10 monclient(hunting): my global_id is 4640 2024-03-29T11:29:59.423 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.423+0000 7f95e3fff640 1 -- 172.21.15.104:0/2250630968 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95f00cbf20 con 0x7f95f00cf820 2024-03-29T11:29:59.423 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.423+0000 7f95e3fff640 1 -- 172.21.15.104:0/2250630968 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 104859838 0 0) 0x7f95dc004eb0 con 0x7f95f00c92b0 2024-03-29T11:29:59.423 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.424+0000 7f95e3fff640 1 -- 172.21.15.104:0/2250630968 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95f002d3d0 con 0x7f95f00c92b0 2024-03-29T11:29:59.423 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.424+0000 7f95e3fff640 1 -- 172.21.15.104:0/2250630968 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f95e40017d0 con 0x7f95f00c92b0 2024-03-29T11:29:59.424 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.424+0000 7f95e3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:59.424 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.424+0000 7f95e3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:59.424 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.424+0000 7f95e3fff640 10 monclient(hunting): dump: 2024-03-29T11:29:59.424 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:29:59.424 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:29:59.424 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:59.424 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:59.424 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:59.424 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:59.424 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:29:59.424 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:29:59.424 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:29:59.424 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:59.424 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.424+0000 7f95e3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:59.425 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.424+0000 7f95e3fff640 1 -- 172.21.15.104:0/2250630968 <== mon.0 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 922075562 0 0) 0x7f95f002d3d0 con 0x7f95f00c92b0 2024-03-29T11:29:59.425 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.424+0000 7f95e3fff640 1 -- 172.21.15.104:0/2250630968 >> v1:172.21.15.104:6790/0 conn(0x7f95f00cf820 legacy=0x7f95f00c0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:59.425 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.424+0000 7f95e3fff640 1 -- 172.21.15.104:0/2250630968 >> v1:172.21.15.119:6789/0 conn(0x7f95f00066e0 legacy=0x7f95f0160a80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:29:59.425 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.424+0000 7f95e3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:29:59.425 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.424+0000 7f95e3fff640 1 monclient: found mon.a 2024-03-29T11:29:59.425 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.424+0000 7f95e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:59.425 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.424+0000 7f95e3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:29:59.425 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.424+0000 7f95e3fff640 1 -- 172.21.15.104:0/2250630968 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95f00ce860 con 0x7f95f00c92b0 2024-03-29T11:29:59.425 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.424+0000 7f95e3fff640 10 monclient: _finish_auth 0 2024-03-29T11:29:59.425 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.424+0000 7f95e3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:29:59.425 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.424+0000 7f95e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:59.425 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.424+0000 7f95fcd41640 5 monclient: authenticate success, global_id 4581 2024-03-29T11:29:59.425 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.424+0000 7f95fcd41640 10 monclient: _renew_subs 2024-03-29T11:29:59.425 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.424+0000 7f95fcd41640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:29:59.425 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.424+0000 7f95fcd41640 1 -- 172.21.15.104:0/2250630968 --> v1:172.21.15.104:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95f0161190 con 0x7f95f00c92b0 2024-03-29T11:29:59.426 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.424+0000 7f95fcd41640 10 client.4581.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:59.426 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.424+0000 7f95fcd41640 10 monclient: _renew_subs 2024-03-29T11:29:59.426 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.424+0000 7f95fcd41640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:29:59.426 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.424+0000 7f95fcd41640 1 -- 172.21.15.104:0/2250630968 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f95f00cc590 con 0x7f95f00c92b0 2024-03-29T11:29:59.426 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.424+0000 7f95fcd41640 1 librados: init done 2024-03-29T11:29:59.426 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.425+0000 7f95e3fff640 1 -- 172.21.15.104:0/2250630968 <== mon.0 v1:172.21.15.104:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f95e4001930 con 0x7f95f00c92b0 2024-03-29T11:29:59.426 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.425+0000 7f95e3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:59.426 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.425+0000 7f95e3fff640 1 -- 172.21.15.104:0/2250630968 <== mon.0 v1:172.21.15.104:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f95e4004940 con 0x7f95f00c92b0 2024-03-29T11:29:59.426 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.425+0000 7f95e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:59.426 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.425+0000 7f95e3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:59.426 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.425+0000 7f95e3fff640 10 monclient: dump: 2024-03-29T11:29:59.426 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:29:59.426 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:29:59.426 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:59.427 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:29:59.427 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:59.427 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:29:59.427 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:29:59.427 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:29:59.427 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:29:59.427 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:29:59.427 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.425+0000 7f95f5ffb640 4 set_mon_vals no callback set 2024-03-29T11:29:59.427 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.426+0000 7f95fcd41640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:59.427 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.426+0000 7f95fcd41640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:59.427 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.426+0000 7f95fcd41640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:29:59.428 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.426+0000 7f95fcd41640 1 -- 172.21.15.104:0/2250630968 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95d8005080 con 0x7f95f00c92b0 2024-03-29T11:29:59.428 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.427+0000 7f95e3fff640 1 -- 172.21.15.104:0/2250630968 <== mon.0 v1:172.21.15.104:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f95e401a910 con 0x7f95f00c92b0 2024-03-29T11:29:59.428 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.429+0000 7f95e3fff640 1 -- 172.21.15.104:0/2250630968 <== mon.0 v1:172.21.15.104:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 668040924 0 0) 0x7f95e4084910 con 0x7f95f00c92b0 2024-03-29T11:29:59.428 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.429+0000 7f95e3fff640 10 client.4581.objecter ms_dispatch 0x7f95f8000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:29:59.428 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.429+0000 7f95e3fff640 3 client.4581.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:29:59.428 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.429+0000 7f95e3fff640 3 client.4581.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:29:59.428 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.429+0000 7f95e3fff640 20 client.4581.objecter dump_active .. 0 homeless 2024-03-29T11:29:59.429 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.431+0000 7f95f77fe640 10 monclient: get_auth_request con 0x7f95dc06dd30 auth_method 0 2024-03-29T11:29:59.429 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.431+0000 7f95e3fff640 10 client.4581.objecter ms_handle_connect 0x7f95dc06dd30 2024-03-29T11:29:59.429 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.435+0000 7f95e3fff640 1 -- 172.21.15.104:0/2250630968 <== mon.0 v1:172.21.15.104:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f95e4050020 con 0x7f95f00c92b0 2024-03-29T11:29:59.429 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.435+0000 7f95e3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:59.429 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.435+0000 7f95e3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:59.754 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:29:59.754 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":11,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:29:29.328850+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:29.328850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:59.754 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.761+0000 7f95fcd41640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:59.754 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.761+0000 7f95fcd41640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:59.754 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.761+0000 7f95fcd41640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:29:59.754 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.761+0000 7f95fcd41640 1 -- 172.21.15.104:0/2250630968 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f95d8005330 con 0x7f95f00c92b0 2024-03-29T11:29:59.755 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.761+0000 7f95e3fff640 1 -- 172.21.15.104:0/2250630968 <== mon.0 v1:172.21.15.104:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3673681244) 0x7f95e4052120 con 0x7f95f00c92b0 2024-03-29T11:29:59.755 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.761+0000 7f95e3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:59.755 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.761+0000 7f95e3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:59.760 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.764+0000 7f95fcd41640 10 librados: watch_flush enter 2024-03-29T11:29:59.760 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.764+0000 7f95fcd41640 10 librados: watch_flush exit 2024-03-29T11:29:59.760 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.764+0000 7f95fcd41640 20 client.4581.objecter shutdown clearing up homeless session... 2024-03-29T11:29:59.760 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.764+0000 7f95fcd41640 10 client.4581.objecter successfully canceled tick 2024-03-29T11:29:59.760 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.767+0000 7f95fcd41640 1 -- 172.21.15.104:0/2250630968 >> v1:172.21.15.104:6816/3341126660 conn(0x7f95dc06dd30 legacy=0x7f95dc0701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:59.760 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.767+0000 7f95fcd41640 10 monclient: shutdown 2024-03-29T11:29:59.760 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.767+0000 7f95fcd41640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:59.760 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.767+0000 7f95fcd41640 1 -- 172.21.15.104:0/2250630968 >> v1:172.21.15.104:6789/0 conn(0x7f95f00c92b0 legacy=0x7f95f00c04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:59.760 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.767+0000 7f95fcd41640 1 -- 172.21.15.104:0/2250630968 shutdown_connections 2024-03-29T11:29:59.760 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.767+0000 7f95fcd41640 1 -- 172.21.15.104:0/2250630968 >> 172.21.15.104:0/2250630968 conn(0x7f95f00058f0 msgr2=0x7f95f00c4ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:59.761 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.767+0000 7f95fcd41640 1 -- 172.21.15.104:0/2250630968 shutdown_connections 2024-03-29T11:29:59.761 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.768+0000 7f95fcd41640 1 -- 172.21.15.104:0/2250630968 wait complete. 2024-03-29T11:29:59.761 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:29:59.768+0000 7f95fcd41640 1 librados: shutdown 2024-03-29T11:29:59.780 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:06.586 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:06.734 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.740+0000 7fad99dcf640 10 monclient: get_monmap_and_config 2024-03-29T11:30:06.734 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.740+0000 7fad99dcf640 10 monclient: build_initial_monmap 2024-03-29T11:30:06.734 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.740+0000 7fad99dcf640 10 monclient: monmap: 2024-03-29T11:30:06.734 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:30:06.734 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:30:06.734 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:30:06.742672+0000 2024-03-29T11:30:06.734 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:30:06.742672+0000 2024-03-29T11:30:06.734 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:06.734 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:30:06.735 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:30:06.735 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:30:06.735 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:30:06.735 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:30:06.735 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.742+0000 7fad99dcf640 1 Processor -- start 2024-03-29T11:30:06.735 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.742+0000 7fad99dcf640 1 -- start start 2024-03-29T11:30:06.735 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.742+0000 7fad99dcf640 10 monclient: init 2024-03-29T11:30:06.736 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.743+0000 7fad99dcf640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:06.737 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.743+0000 7fad99dcf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:30:06.737 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.743+0000 7fad99dcf640 10 monclient(hunting): picked mon.noname-a con 0x7fad940b0860 addr v1:172.21.15.104:6789/0 2024-03-29T11:30:06.737 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.743+0000 7fad99dcf640 10 monclient(hunting): picked mon.noname-b con 0x7fad94072450 addr v1:172.21.15.104:6790/0 2024-03-29T11:30:06.737 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.743+0000 7fad99dcf640 10 monclient(hunting): picked mon.noname-c con 0x7fad9413ee40 addr v1:172.21.15.119:6789/0 2024-03-29T11:30:06.737 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.743+0000 7fad99dcf640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad940ad840 con 0x7fad940b0860 2024-03-29T11:30:06.737 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.743+0000 7fad99dcf640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad9413c830 con 0x7fad9413ee40 2024-03-29T11:30:06.737 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.743+0000 7fad99dcf640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad9413c9c0 con 0x7fad94072450 2024-03-29T11:30:06.737 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.743+0000 7fad99dcf640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:06.737 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.743+0000 7fad99dcf640 10 monclient(hunting): authenticate will time out at 1587.995993s 2024-03-29T11:30:06.738 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.744+0000 7fad937fe640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fad940b0860 0x7fad940acfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:35476/0 (socket says 172.21.15.104:35476) 2024-03-29T11:30:06.738 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.744+0000 7fad937fe640 1 -- 172.21.15.104:0/3821349969 learned_addr learned my addr 172.21.15.104:0/3821349969 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:30:06.738 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.745+0000 7fad927fc640 1 -- 172.21.15.104:0/3821349969 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1985174469 0 0) 0x7fad9413c830 con 0x7fad9413ee40 2024-03-29T11:30:06.738 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.745+0000 7fad927fc640 10 monclient(hunting): _negotiate 2024-03-29T11:30:06.738 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.745+0000 7fad927fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:06.738 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.745+0000 7fad927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:06.738 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.745+0000 7fad927fc640 10 monclient(hunting): my global_id is 4585 2024-03-29T11:30:06.738 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.745+0000 7fad927fc640 1 -- 172.21.15.104:0/3821349969 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad7c003520 con 0x7fad9413ee40 2024-03-29T11:30:06.739 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.745+0000 7fad927fc640 1 -- 172.21.15.104:0/3821349969 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1558562367 0 0) 0x7fad9413c9c0 con 0x7fad94072450 2024-03-29T11:30:06.739 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.745+0000 7fad927fc640 10 monclient(hunting): _negotiate 2024-03-29T11:30:06.739 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.745+0000 7fad927fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:06.739 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.745+0000 7fad927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:06.739 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.745+0000 7fad927fc640 10 monclient(hunting): my global_id is 4643 2024-03-29T11:30:06.739 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.745+0000 7fad927fc640 1 -- 172.21.15.104:0/3821349969 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad9413c830 con 0x7fad94072450 2024-03-29T11:30:06.739 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.745+0000 7fad927fc640 1 -- 172.21.15.104:0/3821349969 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1465065553 0 0) 0x7fad7c003520 con 0x7fad9413ee40 2024-03-29T11:30:06.739 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.745+0000 7fad927fc640 1 -- 172.21.15.104:0/3821349969 >> v1:172.21.15.104:6790/0 conn(0x7fad94072450 legacy=0x7fad9413c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:06.739 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.745+0000 7fad927fc640 1 -- 172.21.15.104:0/3821349969 >> v1:172.21.15.104:6789/0 conn(0x7fad940b0860 legacy=0x7fad940acfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:06.739 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.745+0000 7fad927fc640 10 monclient: _finish_hunting 0 2024-03-29T11:30:06.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.745+0000 7fad927fc640 1 monclient: found mon.noname-c 2024-03-29T11:30:06.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.745+0000 7fad927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:06.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.745+0000 7fad927fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.119:6789/0 2024-03-29T11:30:06.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.745+0000 7fad927fc640 1 -- 172.21.15.104:0/3821349969 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad9413f970 con 0x7fad9413ee40 2024-03-29T11:30:06.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.745+0000 7fad927fc640 10 monclient: _finish_auth 0 2024-03-29T11:30:06.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.745+0000 7fad927fc640 10 monclient: _check_auth_tickets 2024-03-29T11:30:06.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.745+0000 7fad927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:06.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.745+0000 7fad927fc640 1 -- 172.21.15.104:0/3821349969 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fad88001690 con 0x7fad9413ee40 2024-03-29T11:30:06.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.745+0000 7fad927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:06.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.745+0000 7fad927fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:30:06.740 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.745+0000 7fad927fc640 10 monclient: dump: 2024-03-29T11:30:06.740 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:30:06.740 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:30:06.740 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:30:06.741 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:30:06.741 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:06.741 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:30:06.741 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:30:06.741 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:30:06.741 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:30:06.741 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:30:06.741 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.745+0000 7fad99dcf640 5 monclient: authenticate success, global_id 4585 2024-03-29T11:30:06.741 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.745+0000 7fad99dcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:06.742 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.746+0000 7fad927fc640 1 -- 172.21.15.104:0/3821349969 <== mon.1 v1:172.21.15.119:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad880017f0 con 0x7fad9413ee40 2024-03-29T11:30:06.742 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.746+0000 7fad927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:06.742 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.746+0000 7fad927fc640 1 -- 172.21.15.104:0/3821349969 <== mon.1 v1:172.21.15.119:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fad88001c00 con 0x7fad9413ee40 2024-03-29T11:30:06.742 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.746+0000 7fad927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:06.742 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.746+0000 7fad927fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:30:06.742 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.746+0000 7fad927fc640 10 monclient: dump: 2024-03-29T11:30:06.742 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:30:06.742 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:30:06.742 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:30:06.742 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:30:06.742 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:06.743 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:30:06.743 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:30:06.743 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:30:06.743 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:30:06.743 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:30:06.743 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.746+0000 7fad99dcf640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:06.744 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.746+0000 7fad99dcf640 4 set_mon_vals no callback set 2024-03-29T11:30:06.744 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.747+0000 7fad99dcf640 10 monclient: shutdown 2024-03-29T11:30:06.744 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.747+0000 7fad99dcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:06.744 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.747+0000 7fad99dcf640 1 -- 172.21.15.104:0/3821349969 >> v1:172.21.15.119:6789/0 conn(0x7fad9413ee40 legacy=0x7fad9413f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:06.744 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.747+0000 7fad99dcf640 1 -- 172.21.15.104:0/3821349969 shutdown_connections 2024-03-29T11:30:06.744 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.747+0000 7fad99dcf640 1 -- 172.21.15.104:0/3821349969 >> 172.21.15.104:0/3821349969 conn(0x7fad9406b660 msgr2=0x7fad9406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:06.744 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.748+0000 7fad99dcf640 1 -- 172.21.15.104:0/3821349969 shutdown_connections 2024-03-29T11:30:06.744 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.748+0000 7fad99dcf640 1 -- 172.21.15.104:0/3821349969 wait complete. 2024-03-29T11:30:06.752 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.749+0000 7fad99dcf640 10 monclient: build_initial_monmap 2024-03-29T11:30:06.753 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.749+0000 7fad99dcf640 10 monclient: monmap: 2024-03-29T11:30:06.753 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:30:06.753 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:06.753 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:30:06.753 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:30:06.753 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:06.753 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:30:06.753 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:30:06.753 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:30:06.753 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:30:06.753 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:30:06.753 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.749+0000 7fad99dcf640 1 librados: starting msgr at 2024-03-29T11:30:06.753 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.749+0000 7fad99dcf640 1 librados: starting objecter 2024-03-29T11:30:06.754 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.749+0000 7fad99dcf640 1 Processor -- start 2024-03-29T11:30:06.754 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.750+0000 7fad99dcf640 1 -- start start 2024-03-29T11:30:06.754 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.750+0000 7fad99dcf640 1 librados: setting wanted keys 2024-03-29T11:30:06.754 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.750+0000 7fad99dcf640 1 librados: calling monclient init 2024-03-29T11:30:06.754 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.750+0000 7fad99dcf640 10 monclient: init 2024-03-29T11:30:06.754 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.751+0000 7fad99dcf640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:06.754 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.751+0000 7fad99dcf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:30:06.754 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.751+0000 7fad99dcf640 10 monclient(hunting): picked mon.noname-b con 0x7fad94072450 addr v1:172.21.15.119:6789/0 2024-03-29T11:30:06.754 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.751+0000 7fad99dcf640 10 monclient(hunting): picked mon.noname-a con 0x7fad940b0860 addr v1:172.21.15.104:6789/0 2024-03-29T11:30:06.754 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.751+0000 7fad92ffd640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7fad940b0860 0x7fad940b11d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:35992/0 (socket says 172.21.15.104:35992) 2024-03-29T11:30:06.754 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.751+0000 7fad92ffd640 1 -- 172.21.15.104:0/27043914 learned_addr learned my addr 172.21.15.104:0/27043914 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:30:06.754 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.751+0000 7fad99dcf640 10 monclient(hunting): picked mon.noname-c con 0x7fad9413ee40 addr v1:172.21.15.104:6790/0 2024-03-29T11:30:06.757 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.751+0000 7fad99dcf640 1 -- 172.21.15.104:0/27043914 --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad940b3260 con 0x7fad940b0860 2024-03-29T11:30:06.758 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.751+0000 7fad99dcf640 1 -- 172.21.15.104:0/27043914 --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad94134600 con 0x7fad94072450 2024-03-29T11:30:06.758 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.752+0000 7fad99dcf640 1 -- 172.21.15.104:0/27043914 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad940b2410 con 0x7fad9413ee40 2024-03-29T11:30:06.758 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.752+0000 7fad99dcf640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:06.758 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.752+0000 7fad99dcf640 10 monclient(hunting): authenticate will time out at 1588.004111s 2024-03-29T11:30:06.758 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.752+0000 7fad7bfff640 10 client.?.objecter ms_handle_connect 0x7fad940b0860 2024-03-29T11:30:06.758 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.752+0000 7fad7bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:06.758 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.752+0000 7fad7bfff640 1 -- 172.21.15.104:0/27043914 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2069402373 0 0) 0x7fad940b3260 con 0x7fad940b0860 2024-03-29T11:30:06.758 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.752+0000 7fad7bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:30:06.758 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.752+0000 7fad7bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:06.758 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.752+0000 7fad7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:06.758 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.752+0000 7fad7bfff640 10 monclient(hunting): my global_id is 4584 2024-03-29T11:30:06.758 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.752+0000 7fad7bfff640 1 -- 172.21.15.104:0/27043914 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad80004eb0 con 0x7fad940b0860 2024-03-29T11:30:06.758 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.753+0000 7fad7bfff640 10 client.?.objecter ms_handle_connect 0x7fad9413ee40 2024-03-29T11:30:06.758 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.753+0000 7fad7bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:06.759 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.753+0000 7fad7bfff640 10 client.?.objecter ms_handle_connect 0x7fad94072450 2024-03-29T11:30:06.759 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.753+0000 7fad7bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:06.759 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.753+0000 7fad7bfff640 1 -- 172.21.15.104:0/27043914 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 546565848 0 0) 0x7fad94134600 con 0x7fad94072450 2024-03-29T11:30:06.759 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.753+0000 7fad7bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:30:06.759 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.753+0000 7fad7bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:06.759 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.753+0000 7fad7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:06.759 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.753+0000 7fad7bfff640 10 monclient(hunting): my global_id is 4588 2024-03-29T11:30:06.759 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.753+0000 7fad7bfff640 1 -- 172.21.15.104:0/27043914 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad940b3260 con 0x7fad94072450 2024-03-29T11:30:06.759 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.753+0000 7fad7bfff640 1 -- 172.21.15.104:0/27043914 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1438533788 0 0) 0x7fad80004eb0 con 0x7fad940b0860 2024-03-29T11:30:06.759 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.753+0000 7fad7bfff640 1 -- 172.21.15.104:0/27043914 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad94134600 con 0x7fad940b0860 2024-03-29T11:30:06.759 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.753+0000 7fad7bfff640 1 -- 172.21.15.104:0/27043914 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1877071169 0 0) 0x7fad940b2410 con 0x7fad9413ee40 2024-03-29T11:30:06.759 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.753+0000 7fad7bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:30:06.759 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.754+0000 7fad7bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:06.759 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.754+0000 7fad7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:06.760 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.754+0000 7fad7bfff640 10 monclient(hunting): my global_id is 4646 2024-03-29T11:30:06.760 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.754+0000 7fad7bfff640 1 -- 172.21.15.104:0/27043914 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad80004eb0 con 0x7fad9413ee40 2024-03-29T11:30:06.760 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.754+0000 7fad7bfff640 1 -- 172.21.15.104:0/27043914 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fad840019f0 con 0x7fad940b0860 2024-03-29T11:30:06.760 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.754+0000 7fad7bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:06.760 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.754+0000 7fad7bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:30:06.760 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.754+0000 7fad7bfff640 10 monclient(hunting): dump: 2024-03-29T11:30:06.760 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:30:06.760 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:30:06.760 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:30:06.760 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:30:06.760 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:06.760 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:30:06.760 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:30:06.760 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:30:06.760 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:30:06.761 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:30:06.761 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.754+0000 7fad7bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:06.761 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.754+0000 7fad7bfff640 1 -- 172.21.15.104:0/27043914 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1176420372 0 0) 0x7fad940b3260 con 0x7fad94072450 2024-03-29T11:30:06.761 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.754+0000 7fad7bfff640 1 -- 172.21.15.104:0/27043914 --> v1:172.21.15.119:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad940b2410 con 0x7fad94072450 2024-03-29T11:30:06.761 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.754+0000 7fad7bfff640 1 -- 172.21.15.104:0/27043914 <== mon.0 v1:172.21.15.104:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 17796342 0 0) 0x7fad94134600 con 0x7fad940b0860 2024-03-29T11:30:06.761 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.754+0000 7fad7bfff640 1 -- 172.21.15.104:0/27043914 >> v1:172.21.15.104:6790/0 conn(0x7fad9413ee40 legacy=0x7fad940b18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:06.761 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.754+0000 7fad7bfff640 1 -- 172.21.15.104:0/27043914 >> v1:172.21.15.119:6789/0 conn(0x7fad94072450 legacy=0x7fad940b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:06.761 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.754+0000 7fad7bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:30:06.761 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.755+0000 7fad7bfff640 1 monclient: found mon.a 2024-03-29T11:30:06.761 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.755+0000 7fad7bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:06.761 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.755+0000 7fad7bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:30:06.761 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.755+0000 7fad7bfff640 1 -- 172.21.15.104:0/27043914 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad94143090 con 0x7fad940b0860 2024-03-29T11:30:06.761 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.755+0000 7fad7bfff640 10 monclient: _finish_auth 0 2024-03-29T11:30:06.761 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.755+0000 7fad7bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:30:06.761 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.755+0000 7fad7bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:06.762 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.755+0000 7fad99dcf640 5 monclient: authenticate success, global_id 4584 2024-03-29T11:30:06.762 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.755+0000 7fad99dcf640 10 monclient: _renew_subs 2024-03-29T11:30:06.762 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.755+0000 7fad99dcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:30:06.762 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.755+0000 7fad99dcf640 1 -- 172.21.15.104:0/27043914 --> v1:172.21.15.104:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad940b1ff0 con 0x7fad940b0860 2024-03-29T11:30:06.762 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.755+0000 7fad99dcf640 10 client.4584.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:06.762 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.755+0000 7fad99dcf640 10 monclient: _renew_subs 2024-03-29T11:30:06.762 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.755+0000 7fad99dcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:30:06.762 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.755+0000 7fad99dcf640 1 -- 172.21.15.104:0/27043914 --> v1:172.21.15.104:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fad94071940 con 0x7fad940b0860 2024-03-29T11:30:06.762 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.755+0000 7fad99dcf640 1 librados: init done 2024-03-29T11:30:06.762 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.755+0000 7fad7bfff640 1 -- 172.21.15.104:0/27043914 <== mon.0 v1:172.21.15.104:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad84001b50 con 0x7fad940b0860 2024-03-29T11:30:06.762 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.755+0000 7fad7bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:06.762 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.755+0000 7fad7bfff640 1 -- 172.21.15.104:0/27043914 <== mon.0 v1:172.21.15.104:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7fad840038c0 con 0x7fad940b0860 2024-03-29T11:30:06.762 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.755+0000 7fad7bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:06.762 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.755+0000 7fad7bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:06.763 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.755+0000 7fad7bfff640 10 monclient: dump: 2024-03-29T11:30:06.763 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:30:06.763 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:30:06.763 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:30:06.763 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:30:06.763 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:06.763 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:30:06.763 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:30:06.763 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:30:06.763 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:30:06.763 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:30:06.763 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.755+0000 7fad917fa640 4 set_mon_vals no callback set 2024-03-29T11:30:06.763 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.757+0000 7fad99dcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:06.763 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.757+0000 7fad99dcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:06.763 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.757+0000 7fad99dcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:30:06.764 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.757+0000 7fad99dcf640 1 -- 172.21.15.104:0/27043914 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad74005080 con 0x7fad940b0860 2024-03-29T11:30:06.765 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.760+0000 7fad7bfff640 1 -- 172.21.15.104:0/27043914 <== mon.0 v1:172.21.15.104:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7fad84003a20 con 0x7fad940b0860 2024-03-29T11:30:06.765 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.761+0000 7fad7bfff640 1 -- 172.21.15.104:0/27043914 <== mon.0 v1:172.21.15.104:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 668040924 0 0) 0x7fad84084a30 con 0x7fad940b0860 2024-03-29T11:30:06.765 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.761+0000 7fad7bfff640 10 client.4584.objecter ms_dispatch 0x7fad94000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:30:06.765 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.761+0000 7fad7bfff640 3 client.4584.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:30:06.765 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.761+0000 7fad7bfff640 3 client.4584.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:30:06.765 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.761+0000 7fad7bfff640 20 client.4584.objecter dump_active .. 0 homeless 2024-03-29T11:30:06.765 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.761+0000 7fad937fe640 10 monclient: get_auth_request con 0x7fad8006e060 auth_method 0 2024-03-29T11:30:06.765 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.762+0000 7fad7bfff640 10 client.4584.objecter ms_handle_connect 0x7fad8006e060 2024-03-29T11:30:06.765 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.768+0000 7fad7bfff640 1 -- 172.21.15.104:0/27043914 <== mon.0 v1:172.21.15.104:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fad84050140 con 0x7fad940b0860 2024-03-29T11:30:06.765 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.768+0000 7fad7bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:06.766 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:06.768+0000 7fad7bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:07.120 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:07.127+0000 7fad99dcf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:07.120 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:07.127+0000 7fad99dcf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:07.120 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:07.127+0000 7fad99dcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.104:6789/0 2024-03-29T11:30:07.120 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:07.127+0000 7fad99dcf640 1 -- 172.21.15.104:0/27043914 --> v1:172.21.15.104:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad74005330 con 0x7fad940b0860 2024-03-29T11:30:07.121 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:07.128+0000 7fad7bfff640 1 -- 172.21.15.104:0/27043914 <== mon.0 v1:172.21.15.104:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3673681244) 0x7fad84052240 con 0x7fad940b0860 2024-03-29T11:30:07.121 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:07.128+0000 7fad7bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:07.121 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:07.128+0000 7fad7bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:07.121 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:30:07.121 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":11,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:29:29.328850+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:29.328850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:07.125 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:07.131+0000 7fad99dcf640 10 librados: watch_flush enter 2024-03-29T11:30:07.125 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:07.131+0000 7fad99dcf640 10 librados: watch_flush exit 2024-03-29T11:30:07.125 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:07.131+0000 7fad99dcf640 20 client.4584.objecter shutdown clearing up homeless session... 2024-03-29T11:30:07.125 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:07.131+0000 7fad99dcf640 10 client.4584.objecter successfully canceled tick 2024-03-29T11:30:07.125 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:07.132+0000 7fad99dcf640 1 -- 172.21.15.104:0/27043914 >> v1:172.21.15.104:6816/3341126660 conn(0x7fad8006e060 legacy=0x7fad80070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:07.125 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:07.132+0000 7fad99dcf640 10 monclient: shutdown 2024-03-29T11:30:07.125 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:07.132+0000 7fad99dcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:07.125 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:07.132+0000 7fad99dcf640 1 -- 172.21.15.104:0/27043914 >> v1:172.21.15.104:6789/0 conn(0x7fad940b0860 legacy=0x7fad940b11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:07.125 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:07.132+0000 7fad99dcf640 1 -- 172.21.15.104:0/27043914 shutdown_connections 2024-03-29T11:30:07.125 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:07.132+0000 7fad99dcf640 1 -- 172.21.15.104:0/27043914 >> 172.21.15.104:0/27043914 conn(0x7fad9406b660 msgr2=0x7fad9413bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:07.126 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:07.132+0000 7fad99dcf640 1 -- 172.21.15.104:0/27043914 shutdown_connections 2024-03-29T11:30:07.126 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:07.132+0000 7fad99dcf640 1 -- 172.21.15.104:0/27043914 wait complete. 2024-03-29T11:30:07.126 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:07.132+0000 7fad99dcf640 1 librados: shutdown 2024-03-29T11:30:07.140 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:13.949 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:14.096 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.103+0000 7f1e3ed4d640 10 monclient: get_monmap_and_config 2024-03-29T11:30:14.097 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.103+0000 7f1e3ed4d640 10 monclient: build_initial_monmap 2024-03-29T11:30:14.097 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.103+0000 7f1e3ed4d640 10 monclient: monmap: 2024-03-29T11:30:14.097 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:30:14.097 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:30:14.097 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:30:14.104304+0000 2024-03-29T11:30:14.097 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:30:14.104304+0000 2024-03-29T11:30:14.097 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:14.097 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:30:14.097 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:30:14.097 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:30:14.097 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:30:14.097 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:30:14.098 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.104+0000 7f1e3ed4d640 1 Processor -- start 2024-03-29T11:30:14.098 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.104+0000 7f1e3ed4d640 1 -- start start 2024-03-29T11:30:14.098 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.104+0000 7f1e3ed4d640 10 monclient: init 2024-03-29T11:30:14.098 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.105+0000 7f1e3ed4d640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:14.098 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.105+0000 7f1e3ed4d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:30:14.098 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.105+0000 7f1e3ed4d640 10 monclient(hunting): picked mon.noname-c con 0x7f1e380b4100 addr v1:172.21.15.119:6789/0 2024-03-29T11:30:14.098 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.105+0000 7f1e3ed4d640 10 monclient(hunting): picked mon.noname-b con 0x7f1e3813ee90 addr v1:172.21.15.104:6790/0 2024-03-29T11:30:14.099 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.106+0000 7f1e3ed4d640 10 monclient(hunting): picked mon.noname-a con 0x7f1e380af900 addr v1:172.21.15.104:6789/0 2024-03-29T11:30:14.099 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.106+0000 7f1e3ed4d640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e380afd10 con 0x7f1e380af900 2024-03-29T11:30:14.099 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.106+0000 7f1e3ed4d640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e380af6f0 con 0x7f1e380b4100 2024-03-29T11:30:14.099 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.106+0000 7f1e3ed4d640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e380aac80 con 0x7f1e3813ee90 2024-03-29T11:30:14.099 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.106+0000 7f1e3ed4d640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:14.099 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.106+0000 7f1e3ed4d640 10 monclient(hunting): authenticate will time out at 1595.357513s 2024-03-29T11:30:14.099 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.106+0000 7f1e37fff640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f1e3813ee90 0x7f1e3813b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:56698/0 (socket says 172.21.15.104:56698) 2024-03-29T11:30:14.099 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.106+0000 7f1e37fff640 1 -- 172.21.15.104:0/1240077010 learned_addr learned my addr 172.21.15.104:0/1240077010 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:30:14.102 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.109+0000 7f1e377fe640 1 -- 172.21.15.104:0/1240077010 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 346890135 0 0) 0x7f1e380af6f0 con 0x7f1e380b4100 2024-03-29T11:30:14.102 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.109+0000 7f1e377fe640 10 monclient(hunting): _negotiate 2024-03-29T11:30:14.102 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.109+0000 7f1e377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:14.102 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.109+0000 7f1e377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:14.102 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.109+0000 7f1e377fe640 10 monclient(hunting): my global_id is 4591 2024-03-29T11:30:14.102 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.109+0000 7f1e377fe640 1 -- 172.21.15.104:0/1240077010 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e20003520 con 0x7f1e380b4100 2024-03-29T11:30:14.104 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.110+0000 7f1e377fe640 1 -- 172.21.15.104:0/1240077010 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 950718124 0 0) 0x7f1e20003520 con 0x7f1e380b4100 2024-03-29T11:30:14.104 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.110+0000 7f1e377fe640 1 -- 172.21.15.104:0/1240077010 >> v1:172.21.15.104:6790/0 conn(0x7f1e3813ee90 legacy=0x7f1e3813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:14.105 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.110+0000 7f1e377fe640 1 -- 172.21.15.104:0/1240077010 >> v1:172.21.15.104:6789/0 conn(0x7f1e380af900 legacy=0x7f1e3814be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:14.105 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.110+0000 7f1e377fe640 10 monclient: _finish_hunting 0 2024-03-29T11:30:14.105 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.110+0000 7f1e377fe640 1 monclient: found mon.noname-c 2024-03-29T11:30:14.105 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.110+0000 7f1e377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:14.105 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.110+0000 7f1e377fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.119:6789/0 2024-03-29T11:30:14.105 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.110+0000 7f1e377fe640 1 -- 172.21.15.104:0/1240077010 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e380aaf60 con 0x7f1e380b4100 2024-03-29T11:30:14.105 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.110+0000 7f1e377fe640 10 monclient: _finish_auth 0 2024-03-29T11:30:14.105 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.110+0000 7f1e377fe640 10 monclient: _check_auth_tickets 2024-03-29T11:30:14.105 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.110+0000 7f1e377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:14.105 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.110+0000 7f1e377fe640 1 -- 172.21.15.104:0/1240077010 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f1e2c001690 con 0x7f1e380b4100 2024-03-29T11:30:14.105 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.110+0000 7f1e377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:14.105 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.110+0000 7f1e377fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:30:14.105 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.110+0000 7f1e377fe640 10 monclient: dump: 2024-03-29T11:30:14.105 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:30:14.106 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:30:14.106 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:30:14.106 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:30:14.106 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:14.106 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:30:14.106 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:30:14.106 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:30:14.106 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:30:14.106 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:30:14.106 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.110+0000 7f1e3ed4d640 5 monclient: authenticate success, global_id 4591 2024-03-29T11:30:14.106 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.110+0000 7f1e3ed4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:14.106 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.110+0000 7f1e377fe640 1 -- 172.21.15.104:0/1240077010 <== mon.1 v1:172.21.15.119:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1e2c0017f0 con 0x7f1e380b4100 2024-03-29T11:30:14.106 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.110+0000 7f1e377fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:14.106 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.110+0000 7f1e377fe640 1 -- 172.21.15.104:0/1240077010 <== mon.1 v1:172.21.15.119:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f1e2c001c40 con 0x7f1e380b4100 2024-03-29T11:30:14.106 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.110+0000 7f1e377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:14.107 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.110+0000 7f1e377fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:30:14.107 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.110+0000 7f1e377fe640 10 monclient: dump: 2024-03-29T11:30:14.107 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:30:14.107 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:30:14.107 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:30:14.107 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:30:14.107 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:14.107 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:30:14.107 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:30:14.107 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:30:14.107 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:30:14.107 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:30:14.107 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.110+0000 7f1e3ed4d640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:14.107 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.110+0000 7f1e3ed4d640 4 set_mon_vals no callback set 2024-03-29T11:30:14.107 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.111+0000 7f1e3ed4d640 10 monclient: shutdown 2024-03-29T11:30:14.107 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.111+0000 7f1e3ed4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:14.108 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.111+0000 7f1e3ed4d640 1 -- 172.21.15.104:0/1240077010 >> v1:172.21.15.119:6789/0 conn(0x7f1e380b4100 legacy=0x7f1e380adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:14.108 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.111+0000 7f1e3ed4d640 1 -- 172.21.15.104:0/1240077010 shutdown_connections 2024-03-29T11:30:14.108 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.111+0000 7f1e3ed4d640 1 -- 172.21.15.104:0/1240077010 >> 172.21.15.104:0/1240077010 conn(0x7f1e380b35e0 msgr2=0x7f1e380b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:14.108 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.112+0000 7f1e3ed4d640 1 -- 172.21.15.104:0/1240077010 shutdown_connections 2024-03-29T11:30:14.108 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.112+0000 7f1e3ed4d640 1 -- 172.21.15.104:0/1240077010 wait complete. 2024-03-29T11:30:14.109 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.113+0000 7f1e3ed4d640 10 monclient: build_initial_monmap 2024-03-29T11:30:14.109 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.113+0000 7f1e3ed4d640 10 monclient: monmap: 2024-03-29T11:30:14.109 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:30:14.109 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:14.109 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:30:14.109 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:30:14.109 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:14.109 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:30:14.109 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:30:14.109 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:30:14.109 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:30:14.109 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:30:14.109 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.114+0000 7f1e3ed4d640 1 librados: starting msgr at 2024-03-29T11:30:14.109 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.114+0000 7f1e3ed4d640 1 librados: starting objecter 2024-03-29T11:30:14.110 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.114+0000 7f1e3ed4d640 1 Processor -- start 2024-03-29T11:30:14.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.117+0000 7f1e3ed4d640 1 -- start start 2024-03-29T11:30:14.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.117+0000 7f1e3ed4d640 1 librados: setting wanted keys 2024-03-29T11:30:14.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.117+0000 7f1e3ed4d640 1 librados: calling monclient init 2024-03-29T11:30:14.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.117+0000 7f1e3ed4d640 10 monclient: init 2024-03-29T11:30:14.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.118+0000 7f1e3ed4d640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:14.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.118+0000 7f1e3ed4d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:30:14.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.118+0000 7f1e3ed4d640 10 monclient(hunting): picked mon.noname-b con 0x7f1e380af900 addr v1:172.21.15.119:6789/0 2024-03-29T11:30:14.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.118+0000 7f1e3ed4d640 10 monclient(hunting): picked mon.noname-c con 0x7f1e380b4100 addr v1:172.21.15.104:6790/0 2024-03-29T11:30:14.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.118+0000 7f1e3ed4d640 10 monclient(hunting): picked mon.noname-a con 0x7f1e3813ee90 addr v1:172.21.15.104:6789/0 2024-03-29T11:30:14.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.118+0000 7f1e3ed4d640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e381461d0 con 0x7f1e3813ee90 2024-03-29T11:30:14.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.118+0000 7f1e3ed4d640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e381e3de0 con 0x7f1e380af900 2024-03-29T11:30:14.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.118+0000 7f1e37fff640 1 --1- >> v1:172.21.15.104:6790/0 conn(0x7f1e380b4100 0x7f1e381e0c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.104:6790/0 says I am v1:172.21.15.104:57976/0 (socket says 172.21.15.104:57976) 2024-03-29T11:30:14.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.118+0000 7f1e37fff640 1 -- 172.21.15.104:0/759400922 learned_addr learned my addr 172.21.15.104:0/759400922 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:30:14.113 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.118+0000 7f1e3ed4d640 1 -- 172.21.15.104:0/759400922 --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e381e40c0 con 0x7f1e380b4100 2024-03-29T11:30:14.114 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.118+0000 7f1e3ed4d640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:14.114 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.118+0000 7f1e3ed4d640 10 monclient(hunting): authenticate will time out at 1595.369480s 2024-03-29T11:30:14.114 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.119+0000 7f1e357fa640 10 client.?.objecter ms_handle_connect 0x7f1e380b4100 2024-03-29T11:30:14.114 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.119+0000 7f1e357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:14.114 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.120+0000 7f1e357fa640 1 -- 172.21.15.104:0/759400922 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 58638487 0 0) 0x7f1e381e40c0 con 0x7f1e380b4100 2024-03-29T11:30:14.114 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.120+0000 7f1e357fa640 10 monclient(hunting): _negotiate 2024-03-29T11:30:14.114 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.120+0000 7f1e357fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:14.114 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.120+0000 7f1e357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:14.114 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.120+0000 7f1e357fa640 10 monclient(hunting): my global_id is 4652 2024-03-29T11:30:14.114 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.120+0000 7f1e357fa640 1 -- 172.21.15.104:0/759400922 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e24004eb0 con 0x7f1e380b4100 2024-03-29T11:30:14.115 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.121+0000 7f1e357fa640 1 -- 172.21.15.104:0/759400922 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2438232309 0 0) 0x7f1e24004eb0 con 0x7f1e380b4100 2024-03-29T11:30:14.115 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.121+0000 7f1e357fa640 1 -- 172.21.15.104:0/759400922 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1e381e40c0 con 0x7f1e380b4100 2024-03-29T11:30:14.115 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.121+0000 7f1e357fa640 1 -- 172.21.15.104:0/759400922 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f1e28001810 con 0x7f1e380b4100 2024-03-29T11:30:14.115 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.121+0000 7f1e357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:14.115 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.121+0000 7f1e357fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:30:14.115 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.121+0000 7f1e357fa640 10 monclient(hunting): dump: 2024-03-29T11:30:14.115 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:30:14.115 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:30:14.116 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:30:14.116 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:30:14.116 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:14.116 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:30:14.116 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:30:14.116 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:30:14.116 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:30:14.116 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:30:14.116 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.121+0000 7f1e357fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:14.116 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.121+0000 7f1e357fa640 1 -- 172.21.15.104:0/759400922 <== mon.2 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2402077673 0 0) 0x7f1e381e40c0 con 0x7f1e380b4100 2024-03-29T11:30:14.116 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.121+0000 7f1e357fa640 1 -- 172.21.15.104:0/759400922 >> v1:172.21.15.119:6789/0 conn(0x7f1e380af900 legacy=0x7f1e381e4550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:30:14.118 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.124+0000 7f1e357fa640 1 -- 172.21.15.104:0/759400922 >> v1:172.21.15.104:6789/0 conn(0x7f1e3813ee90 legacy=0x7f1e381e1350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:14.118 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.124+0000 7f1e357fa640 10 monclient: _finish_hunting 0 2024-03-29T11:30:14.118 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.124+0000 7f1e357fa640 1 monclient: found mon.c 2024-03-29T11:30:14.118 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.124+0000 7f1e357fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:14.118 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.124+0000 7f1e357fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:30:14.118 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.124+0000 7f1e357fa640 1 -- 172.21.15.104:0/759400922 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e381e1a60 con 0x7f1e380b4100 2024-03-29T11:30:14.118 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.124+0000 7f1e357fa640 10 monclient: _finish_auth 0 2024-03-29T11:30:14.118 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.124+0000 7f1e357fa640 10 monclient: _check_auth_tickets 2024-03-29T11:30:14.118 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.124+0000 7f1e357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:14.118 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.125+0000 7f1e357fa640 10 client.?.objecter ms_handle_connect 0x7f1e3813ee90 2024-03-29T11:30:14.118 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.125+0000 7f1e357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:14.119 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.125+0000 7f1e3ed4d640 5 monclient: authenticate success, global_id 4652 2024-03-29T11:30:14.119 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.125+0000 7f1e3ed4d640 10 monclient: _renew_subs 2024-03-29T11:30:14.119 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.125+0000 7f1e3ed4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:30:14.119 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.125+0000 7f1e3ed4d640 1 -- 172.21.15.104:0/759400922 --> v1:172.21.15.104:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e381e4cc0 con 0x7f1e380b4100 2024-03-29T11:30:14.119 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.125+0000 7f1e3ed4d640 10 client.4652.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:14.119 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.125+0000 7f1e3ed4d640 10 monclient: _renew_subs 2024-03-29T11:30:14.119 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.125+0000 7f1e3ed4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:30:14.119 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.125+0000 7f1e3ed4d640 1 -- 172.21.15.104:0/759400922 --> v1:172.21.15.104:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e38144e90 con 0x7f1e380b4100 2024-03-29T11:30:14.119 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.125+0000 7f1e3ed4d640 1 librados: init done 2024-03-29T11:30:14.119 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.126+0000 7f1e357fa640 1 -- 172.21.15.104:0/759400922 <== mon.2 v1:172.21.15.104:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1e280019e0 con 0x7f1e380b4100 2024-03-29T11:30:14.119 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.126+0000 7f1e357fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:14.120 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.126+0000 7f1e36ffd640 4 set_mon_vals no callback set 2024-03-29T11:30:14.120 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.126+0000 7f1e357fa640 1 -- 172.21.15.104:0/759400922 <== mon.2 v1:172.21.15.104:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f1e28004a40 con 0x7f1e380b4100 2024-03-29T11:30:14.120 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.126+0000 7f1e357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:14.120 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.126+0000 7f1e357fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:14.120 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.126+0000 7f1e357fa640 10 monclient: dump: 2024-03-29T11:30:14.120 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:30:14.120 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:30:14.120 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:30:14.120 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:30:14.120 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:14.120 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:30:14.120 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:30:14.120 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:30:14.120 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:30:14.120 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:30:14.122 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.128+0000 7f1e357fa640 1 -- 172.21.15.104:0/759400922 <== mon.2 v1:172.21.15.104:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f1e28004ba0 con 0x7f1e380b4100 2024-03-29T11:30:14.122 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.129+0000 7f1e357fa640 1 -- 172.21.15.104:0/759400922 <== mon.2 v1:172.21.15.104:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 668040924 0 0) 0x7f1e28084ae0 con 0x7f1e380b4100 2024-03-29T11:30:14.122 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.129+0000 7f1e357fa640 10 client.4652.objecter ms_dispatch 0x7f1e38000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:30:14.122 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.129+0000 7f1e357fa640 3 client.4652.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:30:14.122 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.129+0000 7f1e357fa640 3 client.4652.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:30:14.122 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.129+0000 7f1e357fa640 20 client.4652.objecter dump_active .. 0 homeless 2024-03-29T11:30:14.122 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.129+0000 7f1e3ed4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:14.122 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.129+0000 7f1e3ed4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:14.122 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.129+0000 7f1e3ed4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:30:14.123 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.129+0000 7f1e3ed4d640 1 -- 172.21.15.104:0/759400922 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e18005080 con 0x7f1e380b4100 2024-03-29T11:30:14.123 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.129+0000 7f1e3cae9640 10 monclient: get_auth_request con 0x7f1e2406dbf0 auth_method 0 2024-03-29T11:30:14.123 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.130+0000 7f1e357fa640 10 client.4652.objecter ms_handle_connect 0x7f1e2406dbf0 2024-03-29T11:30:14.130 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.138+0000 7f1e357fa640 1 -- 172.21.15.104:0/759400922 <== mon.2 v1:172.21.15.104:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1e280501f0 con 0x7f1e380b4100 2024-03-29T11:30:14.131 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.138+0000 7f1e357fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:14.131 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.138+0000 7f1e357fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:14.485 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.493+0000 7f1e3ed4d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:14.485 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.493+0000 7f1e3ed4d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:14.486 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.493+0000 7f1e3ed4d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:30:14.486 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.493+0000 7f1e3ed4d640 1 -- 172.21.15.104:0/759400922 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1e18005330 con 0x7f1e380b4100 2024-03-29T11:30:14.486 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.494+0000 7f1e357fa640 1 -- 172.21.15.104:0/759400922 <== mon.2 v1:172.21.15.104:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3673681244) 0x7f1e280522f0 con 0x7f1e380b4100 2024-03-29T11:30:14.486 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.494+0000 7f1e357fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:14.486 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.494+0000 7f1e357fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:14.487 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:30:14.487 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":11,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:29:29.328850+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:29.328850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:14.490 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.498+0000 7f1e3ed4d640 10 librados: watch_flush enter 2024-03-29T11:30:14.490 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.498+0000 7f1e3ed4d640 10 librados: watch_flush exit 2024-03-29T11:30:14.491 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.498+0000 7f1e3ed4d640 20 client.4652.objecter shutdown clearing up homeless session... 2024-03-29T11:30:14.491 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.498+0000 7f1e3ed4d640 10 client.4652.objecter successfully canceled tick 2024-03-29T11:30:14.491 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.498+0000 7f1e3ed4d640 1 -- 172.21.15.104:0/759400922 >> v1:172.21.15.104:6816/3341126660 conn(0x7f1e2406dbf0 legacy=0x7f1e240700b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:14.491 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.498+0000 7f1e3ed4d640 10 monclient: shutdown 2024-03-29T11:30:14.491 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.498+0000 7f1e3ed4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:14.491 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.498+0000 7f1e3ed4d640 1 -- 172.21.15.104:0/759400922 >> v1:172.21.15.104:6790/0 conn(0x7f1e380b4100 legacy=0x7f1e381e0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:14.491 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.498+0000 7f1e3ed4d640 1 -- 172.21.15.104:0/759400922 shutdown_connections 2024-03-29T11:30:14.491 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.498+0000 7f1e3ed4d640 1 -- 172.21.15.104:0/759400922 >> 172.21.15.104:0/759400922 conn(0x7f1e380b35e0 msgr2=0x7f1e38138c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:14.491 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.499+0000 7f1e3ed4d640 1 -- 172.21.15.104:0/759400922 shutdown_connections 2024-03-29T11:30:14.492 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.499+0000 7f1e3ed4d640 1 -- 172.21.15.104:0/759400922 wait complete. 2024-03-29T11:30:14.492 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:14.499+0000 7f1e3ed4d640 1 librados: shutdown 2024-03-29T11:30:14.505 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:21.312 DEBUG:teuthology.orchestra.run.smithi104:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:21.455 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.463+0000 7f665477a640 10 monclient: get_monmap_and_config 2024-03-29T11:30:21.455 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.463+0000 7f665477a640 10 monclient: build_initial_monmap 2024-03-29T11:30:21.455 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.463+0000 7f665477a640 10 monclient: monmap: 2024-03-29T11:30:21.455 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:30:21.455 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:30:21.456 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:30:21.463964+0000 2024-03-29T11:30:21.456 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:30:21.463964+0000 2024-03-29T11:30:21.456 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:21.456 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:30:21.456 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:30:21.456 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-c 2024-03-29T11:30:21.456 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-b 2024-03-29T11:30:21.456 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:30:21.457 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.463+0000 7f665477a640 1 Processor -- start 2024-03-29T11:30:21.457 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.463+0000 7f665477a640 1 -- start start 2024-03-29T11:30:21.457 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.464+0000 7f665477a640 10 monclient: init 2024-03-29T11:30:21.457 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.465+0000 7f665477a640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:21.457 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.465+0000 7f665477a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:30:21.457 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.465+0000 7f665477a640 10 monclient(hunting): picked mon.noname-b con 0x7f664c1457f0 addr v1:172.21.15.104:6790/0 2024-03-29T11:30:21.458 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.465+0000 7f665477a640 10 monclient(hunting): picked mon.noname-a con 0x7f664c13f650 addr v1:172.21.15.104:6789/0 2024-03-29T11:30:21.458 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.465+0000 7f665477a640 10 monclient(hunting): picked mon.noname-c con 0x7f664c0a9d70 addr v1:172.21.15.119:6789/0 2024-03-29T11:30:21.458 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.465+0000 7f665477a640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f664c0aaef0 con 0x7f664c13f650 2024-03-29T11:30:21.458 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.465+0000 7f665477a640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f664c145bd0 con 0x7f664c0a9d70 2024-03-29T11:30:21.458 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.465+0000 7f665477a640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f664c145eb0 con 0x7f664c1457f0 2024-03-29T11:30:21.458 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.465+0000 7f665477a640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:21.458 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.465+0000 7f665477a640 10 monclient(hunting): authenticate will time out at 1602.716186s 2024-03-29T11:30:21.462 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.468+0000 7f6651d15640 1 --1- >> v1:172.21.15.104:6789/0 conn(0x7f664c13f650 0x7f664c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.104:6789/0 says I am v1:172.21.15.104:59399/0 (socket says 172.21.15.104:59399) 2024-03-29T11:30:21.462 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.468+0000 7f6651d15640 1 -- 172.21.15.104:0/3182225383 learned_addr learned my addr 172.21.15.104:0/3182225383 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:30:21.462 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.468+0000 7f6651514640 1 -- 172.21.15.104:0/3182225383 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3299333093 0 0) 0x7f664c0aaef0 con 0x7f664c13f650 2024-03-29T11:30:21.462 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.468+0000 7f6651514640 10 monclient(hunting): _negotiate 2024-03-29T11:30:21.462 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.468+0000 7f6651514640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:21.462 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.468+0000 7f6651514640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:21.462 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.468+0000 7f6651514640 10 monclient(hunting): my global_id is 4593 2024-03-29T11:30:21.462 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.468+0000 7f6651514640 1 -- 172.21.15.104:0/3182225383 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6638003520 con 0x7f664c13f650 2024-03-29T11:30:21.462 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.468+0000 7f6651514640 1 -- 172.21.15.104:0/3182225383 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 512817175 0 0) 0x7f6638003520 con 0x7f664c13f650 2024-03-29T11:30:21.462 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.468+0000 7f6651514640 1 -- 172.21.15.104:0/3182225383 >> v1:172.21.15.104:6790/0 conn(0x7f664c1457f0 legacy=0x7f664c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:30:21.463 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.468+0000 7f6651514640 1 -- 172.21.15.104:0/3182225383 >> v1:172.21.15.119:6789/0 conn(0x7f664c0a9d70 legacy=0x7f664c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:30:21.463 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.468+0000 7f6651514640 10 monclient: _finish_hunting 0 2024-03-29T11:30:21.463 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.468+0000 7f6651514640 1 monclient: found mon.noname-a 2024-03-29T11:30:21.463 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.468+0000 7f6651514640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:21.463 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.468+0000 7f6651514640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.104:6789/0 2024-03-29T11:30:21.463 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.468+0000 7f6651514640 1 -- 172.21.15.104:0/3182225383 --> v1:172.21.15.104:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f664c146340 con 0x7f664c13f650 2024-03-29T11:30:21.463 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.468+0000 7f6651514640 10 monclient: _finish_auth 0 2024-03-29T11:30:21.463 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.468+0000 7f6651514640 10 monclient: _check_auth_tickets 2024-03-29T11:30:21.463 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.468+0000 7f6651514640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:21.463 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.468+0000 7f6651514640 1 -- 172.21.15.104:0/3182225383 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f663c001690 con 0x7f664c13f650 2024-03-29T11:30:21.463 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.468+0000 7f6651514640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:21.463 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.468+0000 7f6651514640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:30:21.463 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.468+0000 7f6651514640 10 monclient: dump: 2024-03-29T11:30:21.463 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:30:21.463 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:30:21.464 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:30:21.464 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:30:21.464 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:21.464 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:30:21.464 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:30:21.464 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:30:21.464 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:30:21.464 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:30:21.464 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.469+0000 7f6651514640 1 -- 172.21.15.104:0/3182225383 <== mon.0 v1:172.21.15.104:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f663c001860 con 0x7f664c13f650 2024-03-29T11:30:21.464 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.469+0000 7f6651514640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:21.464 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.469+0000 7f6651514640 1 -- 172.21.15.104:0/3182225383 <== mon.0 v1:172.21.15.104:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f663c001cb0 con 0x7f664c13f650 2024-03-29T11:30:21.464 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.469+0000 7f6651514640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:21.464 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.469+0000 7f6651514640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:21.464 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.469+0000 7f6651514640 10 monclient: dump: 2024-03-29T11:30:21.464 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:30:21.465 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:30:21.465 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:30:21.465 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:30:21.465 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:21.465 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:30:21.465 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:30:21.465 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:30:21.465 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:30:21.465 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:30:21.466 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.469+0000 7f665477a640 5 monclient: authenticate success, global_id 4593 2024-03-29T11:30:21.466 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.469+0000 7f665477a640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:21.466 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.469+0000 7f665477a640 4 set_mon_vals no callback set 2024-03-29T11:30:21.468 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.470+0000 7f665477a640 10 monclient: shutdown 2024-03-29T11:30:21.468 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.470+0000 7f665477a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:21.468 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.470+0000 7f665477a640 1 -- 172.21.15.104:0/3182225383 >> v1:172.21.15.104:6789/0 conn(0x7f664c13f650 legacy=0x7f664c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:21.468 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.471+0000 7f665477a640 1 -- 172.21.15.104:0/3182225383 shutdown_connections 2024-03-29T11:30:21.468 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.471+0000 7f665477a640 1 -- 172.21.15.104:0/3182225383 >> 172.21.15.104:0/3182225383 conn(0x7f664c066a00 msgr2=0x7f664c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:21.468 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.471+0000 7f665477a640 1 -- 172.21.15.104:0/3182225383 shutdown_connections 2024-03-29T11:30:21.469 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.471+0000 7f665477a640 1 -- 172.21.15.104:0/3182225383 wait complete. 2024-03-29T11:30:21.469 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.472+0000 7f665477a640 10 monclient: build_initial_monmap 2024-03-29T11:30:21.469 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.472+0000 7f665477a640 10 monclient: monmap: 2024-03-29T11:30:21.469 INFO:teuthology.misc.health.smithi104.stderr:epoch 0 2024-03-29T11:30:21.469 INFO:teuthology.misc.health.smithi104.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:21.469 INFO:teuthology.misc.health.smithi104.stderr:last_changed 0.000000 2024-03-29T11:30:21.469 INFO:teuthology.misc.health.smithi104.stderr:created 0.000000 2024-03-29T11:30:21.469 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:21.469 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:30:21.469 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.noname-a 2024-03-29T11:30:21.469 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.noname-b 2024-03-29T11:30:21.469 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.noname-c 2024-03-29T11:30:21.469 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:30:21.469 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.472+0000 7f665477a640 1 librados: starting msgr at 2024-03-29T11:30:21.470 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.472+0000 7f665477a640 1 librados: starting objecter 2024-03-29T11:30:21.470 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.472+0000 7f665477a640 1 Processor -- start 2024-03-29T11:30:21.470 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.472+0000 7f665477a640 1 -- start start 2024-03-29T11:30:21.470 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.472+0000 7f665477a640 1 librados: setting wanted keys 2024-03-29T11:30:21.470 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.472+0000 7f665477a640 1 librados: calling monclient init 2024-03-29T11:30:21.470 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.472+0000 7f665477a640 10 monclient: init 2024-03-29T11:30:21.470 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.473+0000 7f665477a640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:21.470 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.473+0000 7f665477a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:30:21.470 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.473+0000 7f665477a640 10 monclient(hunting): picked mon.noname-b con 0x7f664c0a9d70 addr v1:172.21.15.119:6789/0 2024-03-29T11:30:21.470 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.473+0000 7f665477a640 10 monclient(hunting): picked mon.noname-c con 0x7f664c13f650 addr v1:172.21.15.104:6790/0 2024-03-29T11:30:21.478 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.475+0000 7f665477a640 10 monclient(hunting): picked mon.noname-a con 0x7f664c1457f0 addr v1:172.21.15.104:6789/0 2024-03-29T11:30:21.478 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.475+0000 7f665477a640 1 -- --> v1:172.21.15.104:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f664c133150 con 0x7f664c1457f0 2024-03-29T11:30:21.479 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.475+0000 7f665477a640 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f664c134b20 con 0x7f664c0a9d70 2024-03-29T11:30:21.479 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.475+0000 7f665477a640 1 -- --> v1:172.21.15.104:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f664c134e00 con 0x7f664c13f650 2024-03-29T11:30:21.479 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.475+0000 7f665477a640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:21.479 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.475+0000 7f665477a640 10 monclient(hunting): authenticate will time out at 1602.726228s 2024-03-29T11:30:21.479 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.478+0000 7f6652516640 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f664c0a9d70 0x7f664c1413c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.104:52890/0 (socket says 172.21.15.104:52890) 2024-03-29T11:30:21.479 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.478+0000 7f6652516640 1 -- 172.21.15.104:0/1041203545 learned_addr learned my addr 172.21.15.104:0/1041203545 (peer_addr_for_me v1:172.21.15.104:0/0) 2024-03-29T11:30:21.479 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.479+0000 7f6642ffd640 10 client.?.objecter ms_handle_connect 0x7f664c1457f0 2024-03-29T11:30:21.479 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.479+0000 7f6642ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:21.479 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.479+0000 7f6642ffd640 1 -- 172.21.15.104:0/1041203545 <== mon.0 v1:172.21.15.104:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4133023842 0 0) 0x7f664c133150 con 0x7f664c1457f0 2024-03-29T11:30:21.479 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.479+0000 7f6642ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:30:21.479 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.479+0000 7f6642ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:21.479 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.479+0000 7f6642ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:21.479 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.479+0000 7f6642ffd640 10 monclient(hunting): my global_id is 4596 2024-03-29T11:30:21.479 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.479+0000 7f6642ffd640 1 -- 172.21.15.104:0/1041203545 --> v1:172.21.15.104:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6630004eb0 con 0x7f664c1457f0 2024-03-29T11:30:21.480 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.479+0000 7f6642ffd640 10 client.?.objecter ms_handle_connect 0x7f664c13f650 2024-03-29T11:30:21.480 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.479+0000 7f6642ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:21.480 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.479+0000 7f6642ffd640 10 client.?.objecter ms_handle_connect 0x7f664c0a9d70 2024-03-29T11:30:21.480 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.479+0000 7f6642ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:21.480 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.479+0000 7f6642ffd640 1 -- 172.21.15.104:0/1041203545 <== mon.2 v1:172.21.15.104:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3945488786 0 0) 0x7f664c134e00 con 0x7f664c13f650 2024-03-29T11:30:21.480 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.479+0000 7f6642ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:30:21.480 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.479+0000 7f6642ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:21.480 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.479+0000 7f6642ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:21.480 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.479+0000 7f6642ffd640 10 monclient(hunting): my global_id is 4655 2024-03-29T11:30:21.480 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.479+0000 7f6642ffd640 1 -- 172.21.15.104:0/1041203545 --> v1:172.21.15.104:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f664c133150 con 0x7f664c13f650 2024-03-29T11:30:21.480 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.480+0000 7f6642ffd640 1 -- 172.21.15.104:0/1041203545 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1004870852 0 0) 0x7f664c134b20 con 0x7f664c0a9d70 2024-03-29T11:30:21.480 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.480+0000 7f6642ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:30:21.480 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.480+0000 7f6642ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:21.480 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.480+0000 7f6642ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:21.481 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.480+0000 7f6642ffd640 10 monclient(hunting): my global_id is 4594 2024-03-29T11:30:21.481 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.480+0000 7f6642ffd640 1 -- 172.21.15.104:0/1041203545 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f664c134e00 con 0x7f664c0a9d70 2024-03-29T11:30:21.481 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.480+0000 7f6642ffd640 1 -- 172.21.15.104:0/1041203545 <== mon.2 v1:172.21.15.104:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 730873108 0 0) 0x7f664c133150 con 0x7f664c13f650 2024-03-29T11:30:21.481 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.480+0000 7f6642ffd640 1 -- 172.21.15.104:0/1041203545 --> v1:172.21.15.104:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f664c134b20 con 0x7f664c13f650 2024-03-29T11:30:21.481 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.480+0000 7f6642ffd640 1 -- 172.21.15.104:0/1041203545 <== mon.0 v1:172.21.15.104:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 544603411 0 0) 0x7f6630004eb0 con 0x7f664c1457f0 2024-03-29T11:30:21.481 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.480+0000 7f6642ffd640 1 -- 172.21.15.104:0/1041203545 --> v1:172.21.15.104:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f664c133150 con 0x7f664c1457f0 2024-03-29T11:30:21.481 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.480+0000 7f6642ffd640 1 -- 172.21.15.104:0/1041203545 <== mon.2 v1:172.21.15.104:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f663c001b70 con 0x7f664c13f650 2024-03-29T11:30:21.481 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.481+0000 7f6642ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:21.481 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.481+0000 7f6642ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:30:21.481 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.481+0000 7f6642ffd640 10 monclient(hunting): dump: 2024-03-29T11:30:21.481 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:30:21.481 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:30:21.481 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:30:21.481 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:30:21.482 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:21.482 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:30:21.482 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:30:21.482 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:30:21.482 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:30:21.482 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:30:21.482 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.481+0000 7f6642ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:21.482 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.481+0000 7f6642ffd640 1 -- 172.21.15.104:0/1041203545 <== mon.0 v1:172.21.15.104:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f6644001af0 con 0x7f664c1457f0 2024-03-29T11:30:21.482 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.481+0000 7f6642ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:21.482 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.481+0000 7f6642ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:21.482 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.481+0000 7f6642ffd640 10 monclient(hunting): dump: 2024-03-29T11:30:21.482 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:30:21.482 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:30:21.482 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:30:21.482 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:30:21.483 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:21.483 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:30:21.483 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:30:21.483 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:30:21.483 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:30:21.483 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:30:21.483 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.481+0000 7f6642ffd640 1 -- 172.21.15.104:0/1041203545 <== mon.2 v1:172.21.15.104:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 214022006 0 0) 0x7f664c134b20 con 0x7f664c13f650 2024-03-29T11:30:21.483 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.481+0000 7f6642ffd640 1 -- 172.21.15.104:0/1041203545 >> v1:172.21.15.119:6789/0 conn(0x7f664c0a9d70 legacy=0x7f664c1413c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:21.483 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.481+0000 7f6642ffd640 1 -- 172.21.15.104:0/1041203545 >> v1:172.21.15.104:6789/0 conn(0x7f664c1457f0 legacy=0x7f664c135c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:21.483 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.481+0000 7f6642ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:30:21.483 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.481+0000 7f6642ffd640 1 monclient: found mon.c 2024-03-29T11:30:21.483 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.481+0000 7f6642ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:21.483 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.481+0000 7f6642ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:30:21.483 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.481+0000 7f6642ffd640 1 -- 172.21.15.104:0/1041203545 --> v1:172.21.15.104:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f664c1350e0 con 0x7f664c13f650 2024-03-29T11:30:21.483 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.481+0000 7f6642ffd640 10 monclient: _finish_auth 0 2024-03-29T11:30:21.484 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.481+0000 7f6642ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:30:21.484 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.481+0000 7f6642ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:21.484 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.481+0000 7f665477a640 5 monclient: authenticate success, global_id 4655 2024-03-29T11:30:21.484 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.481+0000 7f665477a640 10 monclient: _renew_subs 2024-03-29T11:30:21.484 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.481+0000 7f665477a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:30:21.484 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.481+0000 7f665477a640 1 -- 172.21.15.104:0/1041203545 --> v1:172.21.15.104:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f664c133dd0 con 0x7f664c13f650 2024-03-29T11:30:21.484 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.481+0000 7f665477a640 10 client.4655.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:21.484 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.481+0000 7f665477a640 10 monclient: _renew_subs 2024-03-29T11:30:21.484 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.481+0000 7f665477a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:30:21.484 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.481+0000 7f665477a640 1 -- 172.21.15.104:0/1041203545 --> v1:172.21.15.104:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f664c141cb0 con 0x7f664c13f650 2024-03-29T11:30:21.484 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.481+0000 7f665477a640 1 librados: init done 2024-03-29T11:30:21.484 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.482+0000 7f6642ffd640 1 -- 172.21.15.104:0/1041203545 <== mon.2 v1:172.21.15.104:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f663c001cd0 con 0x7f664c13f650 2024-03-29T11:30:21.484 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.482+0000 7f6642ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:21.484 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.482+0000 7f6642ffd640 1 -- 172.21.15.104:0/1041203545 <== mon.2 v1:172.21.15.104:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2296997550 0 0) 0x7f663c004d90 con 0x7f664c13f650 2024-03-29T11:30:21.485 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.482+0000 7f6642ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:21.485 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.482+0000 7f6642ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:21.485 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.482+0000 7f6642ffd640 10 monclient: dump: 2024-03-29T11:30:21.485 INFO:teuthology.misc.health.smithi104.stderr:epoch 1 2024-03-29T11:30:21.485 INFO:teuthology.misc.health.smithi104.stderr:fsid da26e8fd-d28e-492a-aadb-08e647a06b62 2024-03-29T11:30:21.485 INFO:teuthology.misc.health.smithi104.stderr:last_changed 2024-03-29T11:18:46.289619+0000 2024-03-29T11:30:21.485 INFO:teuthology.misc.health.smithi104.stderr:created 2024-03-29T11:18:46.289619+0000 2024-03-29T11:30:21.485 INFO:teuthology.misc.health.smithi104.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:21.485 INFO:teuthology.misc.health.smithi104.stderr:election_strategy: 1 2024-03-29T11:30:21.485 INFO:teuthology.misc.health.smithi104.stderr:0: v1:172.21.15.104:6789/0 mon.a 2024-03-29T11:30:21.485 INFO:teuthology.misc.health.smithi104.stderr:1: v1:172.21.15.119:6789/0 mon.b 2024-03-29T11:30:21.485 INFO:teuthology.misc.health.smithi104.stderr:2: v1:172.21.15.104:6790/0 mon.c 2024-03-29T11:30:21.485 INFO:teuthology.misc.health.smithi104.stderr: 2024-03-29T11:30:21.485 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.482+0000 7f6643fff640 4 set_mon_vals no callback set 2024-03-29T11:30:21.485 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.484+0000 7f6642ffd640 1 -- 172.21.15.104:0/1041203545 <== mon.2 v1:172.21.15.104:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 93729501 0 0) 0x7f663c01ad60 con 0x7f664c13f650 2024-03-29T11:30:21.486 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.484+0000 7f6652516640 10 monclient: get_auth_request con 0x7f663006e080 auth_method 0 2024-03-29T11:30:21.486 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.485+0000 7f665477a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:21.486 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.485+0000 7f665477a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:21.486 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.485+0000 7f665477a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:30:21.486 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.485+0000 7f665477a640 1 -- 172.21.15.104:0/1041203545 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6634005080 con 0x7f664c13f650 2024-03-29T11:30:21.486 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.485+0000 7f6642ffd640 1 -- 172.21.15.104:0/1041203545 <== mon.2 v1:172.21.15.104:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 668040924 0 0) 0x7f663c084a40 con 0x7f664c13f650 2024-03-29T11:30:21.486 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.485+0000 7f6642ffd640 10 client.4655.objecter ms_dispatch 0x7f664c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:30:21.486 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.485+0000 7f6642ffd640 3 client.4655.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:30:21.486 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.485+0000 7f6642ffd640 3 client.4655.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:30:21.486 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.485+0000 7f6642ffd640 20 client.4655.objecter dump_active .. 0 homeless 2024-03-29T11:30:21.486 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.485+0000 7f6642ffd640 10 client.4655.objecter ms_handle_connect 0x7f663006e080 2024-03-29T11:30:21.487 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.493+0000 7f6642ffd640 1 -- 172.21.15.104:0/1041203545 <== mon.2 v1:172.21.15.104:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f663c0503a0 con 0x7f664c13f650 2024-03-29T11:30:21.487 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.493+0000 7f6642ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:21.487 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.494+0000 7f6642ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:21.814 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.821+0000 7f665477a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:21.814 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.821+0000 7f665477a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:21.814 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.821+0000 7f665477a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.104:6790/0 2024-03-29T11:30:21.814 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.821+0000 7f665477a640 1 -- 172.21.15.104:0/1041203545 --> v1:172.21.15.104:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6634005330 con 0x7f664c13f650 2024-03-29T11:30:21.814 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.822+0000 7f6642ffd640 1 -- 172.21.15.104:0/1041203545 <== mon.2 v1:172.21.15.104:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3673681244) 0x7f663c0522d0 con 0x7f664c13f650 2024-03-29T11:30:21.815 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.822+0000 7f6642ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:21.815 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.822+0000 7f6642ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:21.816 INFO:teuthology.misc.health.smithi104.stdout: 2024-03-29T11:30:21.816 INFO:teuthology.misc.health.smithi104.stdout:{"epoch":11,"fsid":"da26e8fd-d28e-492a-aadb-08e647a06b62","created":"2024-03-29T11:19:24.391256+0000","modified":"2024-03-29T11:29:29.328850+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:29.328850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9aa5839f-9605-4fab-b090-d4c7de9f53ec","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"ccf1b3a9-273a-4c32-931c-c7e8df9c6e5c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"9bcd508f-a564-47de-bc8d-d287c8a0441f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"991d3031-9687-404a-a087-a3781ccfdc63","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"010f8945-19c0-4f55-b2fc-21d83b076f4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"123ab874-0241-4d9a-b9ff-e35e7bb09898","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9c0a3b9b-48bb-4cd2-a973-ab112c32f77f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"352d1ad9-6778-43f4-a7e4-5345e7c19de0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:21.819 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.827+0000 7f665477a640 10 librados: watch_flush enter 2024-03-29T11:30:21.820 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.827+0000 7f665477a640 10 librados: watch_flush exit 2024-03-29T11:30:21.820 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.827+0000 7f665477a640 20 client.4655.objecter shutdown clearing up homeless session... 2024-03-29T11:30:21.820 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.827+0000 7f665477a640 10 client.4655.objecter successfully canceled tick 2024-03-29T11:30:21.820 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.828+0000 7f665477a640 1 -- 172.21.15.104:0/1041203545 >> v1:172.21.15.104:6816/3341126660 conn(0x7f663006e080 legacy=0x7f6630070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:21.820 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.828+0000 7f665477a640 10 monclient: shutdown 2024-03-29T11:30:21.820 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.828+0000 7f665477a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:21.820 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.828+0000 7f665477a640 1 -- 172.21.15.104:0/1041203545 >> v1:172.21.15.104:6790/0 conn(0x7f664c13f650 legacy=0x7f664c135500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:21.820 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.828+0000 7f665477a640 1 -- 172.21.15.104:0/1041203545 shutdown_connections 2024-03-29T11:30:21.821 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.828+0000 7f665477a640 1 -- 172.21.15.104:0/1041203545 >> 172.21.15.104:0/1041203545 conn(0x7f664c066a00 msgr2=0x7f664c130a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:21.821 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.828+0000 7f665477a640 1 -- 172.21.15.104:0/1041203545 shutdown_connections 2024-03-29T11:30:21.827 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.832+0000 7f665477a640 1 -- 172.21.15.104:0/1041203545 wait complete. 2024-03-29T11:30:21.827 INFO:teuthology.misc.health.smithi104.stderr:2024-03-29T11:30:21.835+0000 7f665477a640 1 librados: shutdown 2024-03-29T11:30:21.845 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:21.845 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:30:21.847 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T11:30:21.847 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T11:30:21.847 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:21.885 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T11:30:21.885 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T11:30:21.885 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:21.973 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T11:30:21.973 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T11:30:21.974 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:22.010 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T11:30:22.010 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-29T11:30:22.010 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:22.081 INFO:tasks.ceph.osd.3:Stopped 2024-03-29T11:30:22.082 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-29T11:30:22.082 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:22.123 INFO:tasks.ceph.osd.4:Stopped 2024-03-29T11:30:22.123 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-29T11:30:22.123 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:22.199 INFO:tasks.ceph.osd.5:Stopped 2024-03-29T11:30:22.199 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-29T11:30:22.200 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:22.277 INFO:tasks.ceph.osd.6:Stopped 2024-03-29T11:30:22.277 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-29T11:30:22.277 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:22.348 INFO:tasks.ceph.osd.7:Stopped 2024-03-29T11:30:22.348 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T11:30:22.348 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-29T11:30:22.348 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:22.372 INFO:tasks.ceph.mgr.y:Stopped 2024-03-29T11:30:22.373 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T11:30:22.373 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:22.412 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T11:30:22.412 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T11:30:22.412 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T11:30:22.412 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:22.423 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T11:30:22.424 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-29T11:30:22.424 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:22.475 INFO:tasks.ceph.mon.c:Stopped 2024-03-29T11:30:22.476 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-29T11:30:22.476 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:22.487 INFO:tasks.ceph.mon.b:Stopped 2024-03-29T11:30:22.487 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T11:30:22.487 DEBUG:teuthology.orchestra.run.smithi104:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-03-29T11:30:22.554 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi104.front.sepia.ceph.com 2024-03-29T11:30:22.555 DEBUG:teuthology.orchestra.run.smithi104:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T11:30:24.180 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi104.front.sepia.ceph.com 2024-03-29T11:30:24.180 DEBUG:teuthology.orchestra.run.smithi104:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T11:30:24.324 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi104.front.sepia.ceph.com 2024-03-29T11:30:24.325 DEBUG:teuthology.orchestra.run.smithi104:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T11:30:24.383 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi104.front.sepia.ceph.com 2024-03-29T11:30:24.383 DEBUG:teuthology.orchestra.run.smithi104:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-29T11:30:24.448 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi119.front.sepia.ceph.com 2024-03-29T11:30:24.448 DEBUG:teuthology.orchestra.run.smithi119:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-29T11:30:27.519 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi119.front.sepia.ceph.com 2024-03-29T11:30:27.520 DEBUG:teuthology.orchestra.run.smithi119:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-29T11:30:27.596 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi119.front.sepia.ceph.com 2024-03-29T11:30:27.597 DEBUG:teuthology.orchestra.run.smithi119:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-29T11:30:27.669 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi119.front.sepia.ceph.com 2024-03-29T11:30:27.669 DEBUG:teuthology.orchestra.run.smithi119:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-29T11:30:27.720 INFO:tasks.ceph:Archiving mon data... 2024-03-29T11:30:27.721 DEBUG:teuthology.misc:Transferring archived files from smithi104:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622681/data/mon.a.tgz 2024-03-29T11:30:27.721 DEBUG:teuthology.orchestra.run.smithi104:> mktemp 2024-03-29T11:30:27.739 INFO:teuthology.orchestra.run.smithi104.stdout:/tmp/tmp.oroNrAUsv8 2024-03-29T11:30:27.740 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.oroNrAUsv8 2024-03-29T11:30:27.939 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 0666 /tmp/tmp.oroNrAUsv8 2024-03-29T11:30:28.026 DEBUG:teuthology.orchestra.remote:smithi104:/tmp/tmp.oroNrAUsv8 is 211KB 2024-03-29T11:30:28.121 DEBUG:teuthology.orchestra.run.smithi104:> rm -fr /tmp/tmp.oroNrAUsv8 2024-03-29T11:30:28.136 DEBUG:teuthology.misc:Transferring archived files from smithi104:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622681/data/mon.c.tgz 2024-03-29T11:30:28.136 DEBUG:teuthology.orchestra.run.smithi104:> mktemp 2024-03-29T11:30:28.192 INFO:teuthology.orchestra.run.smithi104.stdout:/tmp/tmp.zDmkviHvoK 2024-03-29T11:30:28.192 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.zDmkviHvoK 2024-03-29T11:30:28.293 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 0666 /tmp/tmp.zDmkviHvoK 2024-03-29T11:30:28.378 DEBUG:teuthology.orchestra.remote:smithi104:/tmp/tmp.zDmkviHvoK is 211KB 2024-03-29T11:30:28.472 DEBUG:teuthology.orchestra.run.smithi104:> rm -fr /tmp/tmp.zDmkviHvoK 2024-03-29T11:30:28.486 DEBUG:teuthology.misc:Transferring archived files from smithi119:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622681/data/mon.b.tgz 2024-03-29T11:30:28.487 DEBUG:teuthology.orchestra.run.smithi119:> mktemp 2024-03-29T11:30:28.505 INFO:teuthology.orchestra.run.smithi119.stdout:/tmp/tmp.bND4JKFIqb 2024-03-29T11:30:28.505 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.bND4JKFIqb 2024-03-29T11:30:28.622 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 0666 /tmp/tmp.bND4JKFIqb 2024-03-29T11:30:28.711 DEBUG:teuthology.orchestra.remote:smithi119:/tmp/tmp.bND4JKFIqb is 211KB 2024-03-29T11:30:28.804 DEBUG:teuthology.orchestra.run.smithi119:> rm -fr /tmp/tmp.bND4JKFIqb 2024-03-29T11:30:28.819 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T11:30:28.819 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:30:28.822 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:30:28.901 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:30:28.943 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:30:28.943 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T11:30:28.945 DEBUG:teuthology.misc:Transferring archived files from smithi104:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622681/remote/smithi104/crash 2024-03-29T11:30:28.946 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:30:28.973 DEBUG:teuthology.misc:Transferring archived files from smithi119:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622681/remote/smithi119/crash 2024-03-29T11:30:28.974 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:30:29.009 INFO:tasks.ceph:Compressing logs... 2024-03-29T11:30:29.009 DEBUG:teuthology.orchestra.run.smithi104:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:30:29.018 DEBUG:teuthology.orchestra.run.smithi119:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:30:29.075 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39767.log 2024-03-29T11:30:29.075 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38238.log 2024-03-29T11:30:29.075 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.39767.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T11:30:29.075 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.38238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40508.log 2024-03-29T11:30:29.076 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-mon.a.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39767.log.gz 2024-03-29T11:30:29.076 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41328.log 2024-03-29T11:30:29.076 INFO:teuthology.orchestra.run.smithi104.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.38238.log.gz 2024-03-29T11:30:29.076 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.40508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41289.log 2024-03-29T11:30:29.076 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.41328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41406.log 2024-03-29T11:30:29.076 INFO:teuthology.orchestra.run.smithi104.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40508.log.gz 2024-03-29T11:30:29.076 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.41289.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41328.log.gz 2024-03-29T11:30:29.076 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38632.log 2024-03-29T11:30:29.076 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.41406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40235.log 2024-03-29T11:30:29.076 INFO:teuthology.orchestra.run.smithi104.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41289.log.gz 2024-03-29T11:30:29.076 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.38632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41367.log 2024-03-29T11:30:29.076 INFO:teuthology.orchestra.run.smithi104.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41406.log.gz 2024-03-29T11:30:29.076 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.40235.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38632.log.gz 2024-03-29T11:30:29.076 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38160.log 2024-03-29T11:30:29.077 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.41367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39845.log 2024-03-29T11:30:29.077 INFO:teuthology.orchestra.run.smithi104.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40235.log.gz 2024-03-29T11:30:29.077 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.38160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38277.log 2024-03-29T11:30:29.077 INFO:teuthology.orchestra.run.smithi104.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41367.log.gz 2024-03-29T11:30:29.077 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.39845.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38160.log.gz 2024-03-29T11:30:29.077 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39806.log 2024-03-29T11:30:29.077 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.38277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40664.log 2024-03-29T11:30:29.077 INFO:teuthology.orchestra.run.smithi104.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39845.log.gz 2024-03-29T11:30:29.077 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.39806.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38277.log.gz 2024-03-29T11:30:29.077 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41445.log 2024-03-29T11:30:29.077 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.40664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38710.log 2024-03-29T11:30:29.077 INFO:teuthology.orchestra.run.smithi104.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39806.log.gz 2024-03-29T11:30:29.077 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.41445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39373.log 2024-03-29T11:30:29.077 INFO:teuthology.orchestra.run.smithi104.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40664.log.gz 2024-03-29T11:30:29.077 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.38710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40079.log 2024-03-29T11:30:29.078 INFO:teuthology.orchestra.run.smithi104.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41445.log.gz 2024-03-29T11:30:29.078 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.39373.log: 86.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.38710.log.gz 2024-03-29T11:30:29.078 INFO:teuthology.orchestra.run.smithi104.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39217.log 2024-03-29T11:30:29.078 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.40079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40118.log 2024-03-29T11:30:29.078 INFO:teuthology.orchestra.run.smithi104.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39373.log.gz 2024-03-29T11:30:29.078 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.39217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41211.log 2024-03-29T11:30:29.078 INFO:teuthology.orchestra.run.smithi104.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40079.log.gz 2024-03-29T11:30:29.078 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.40118.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39217.log.gz 2024-03-29T11:30:29.078 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38788.log 2024-03-29T11:30:29.078 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.41211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40820.log 2024-03-29T11:30:29.078 INFO:teuthology.orchestra.run.smithi104.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40118.log.gz 2024-03-29T11:30:29.078 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.38788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39728.log 2024-03-29T11:30:29.078 INFO:teuthology.orchestra.run.smithi104.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41211.log.gz 2024-03-29T11:30:29.078 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.40820.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38788.log.gz 2024-03-29T11:30:29.078 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34651.log 2024-03-29T11:30:29.079 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.39728.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:30:29.079 INFO:teuthology.orchestra.run.smithi104.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40820.log.gz 2024-03-29T11:30:29.079 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.34651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40625.log 2024-03-29T11:30:29.079 INFO:teuthology.orchestra.run.smithi104.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39728.log.gz 2024-03-29T11:30:29.079 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph.audit.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34651.log.gz 2024-03-29T11:30:29.079 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40898.log 2024-03-29T11:30:29.079 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.40625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39688.log 2024-03-29T11:30:29.079 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.40898.log: 93.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:30:29.079 INFO:teuthology.orchestra.run.smithi104.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.40625.log.gz 2024-03-29T11:30:29.079 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41055.log 2024-03-29T11:30:29.079 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40196.log 2024-03-29T11:30:29.079 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.39688.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40898.log.gz 2024-03-29T11:30:29.079 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.41055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40586.log 2024-03-29T11:30:29.079 INFO:teuthology.orchestra.run.smithi104.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39688.log.gz 2024-03-29T11:30:29.080 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.40196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39295.log 2024-03-29T11:30:29.080 INFO:teuthology.orchestra.run.smithi104.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41055.log.gz 2024-03-29T11:30:29.080 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40781.log 2024-03-29T11:30:29.080 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.40586.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40196.log.gz 2024-03-29T11:30:29.080 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.39295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38983.log 2024-03-29T11:30:29.080 INFO:teuthology.orchestra.run.smithi104.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40586.log.gz 2024-03-29T11:30:29.080 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.40781.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39295.log.gz 2024-03-29T11:30:29.080 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40430.log 2024-03-29T11:30:29.080 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.38983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40469.log 2024-03-29T11:30:29.080 INFO:teuthology.orchestra.run.smithi104.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40781.log.gz 2024-03-29T11:30:29.080 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.40430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40001.log 2024-03-29T11:30:29.080 INFO:teuthology.orchestra.run.smithi104.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38983.log.gz 2024-03-29T11:30:29.080 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.40469.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40430.log.gzgzip 2024-03-29T11:30:29.080 INFO:teuthology.orchestra.run.smithi104.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38553.log 2024-03-29T11:30:29.080 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.40001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39884.log 2024-03-29T11:30:29.080 INFO:teuthology.orchestra.run.smithi104.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40469.log.gz 2024-03-29T11:30:29.081 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.38553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41015.log 2024-03-29T11:30:29.081 INFO:teuthology.orchestra.run.smithi104.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40001.log.gz 2024-03-29T11:30:29.081 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.39884.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-29T11:30:29.081 INFO:teuthology.orchestra.run.smithi104.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38553.log.gz 2024-03-29T11:30:29.081 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.41015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41172.log 2024-03-29T11:30:29.081 INFO:teuthology.orchestra.run.smithi104.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39884.log.gz 2024-03-29T11:30:29.081 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38593.log 2024-03-29T11:30:29.081 INFO:teuthology.orchestra.run.smithi104.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41015.log.gz 2024-03-29T11:30:29.081 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.41172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39256.log 2024-03-29T11:30:29.081 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.38593.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41172.log.gz 2024-03-29T11:30:29.081 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38944.log 2024-03-29T11:30:29.081 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.39256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39061.log 2024-03-29T11:30:29.081 INFO:teuthology.orchestra.run.smithi104.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38593.log.gz 2024-03-29T11:30:29.081 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.38944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38866.log 2024-03-29T11:30:29.081 INFO:teuthology.orchestra.run.smithi104.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39256.log.gz 2024-03-29T11:30:29.082 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39412.log 2024-03-29T11:30:29.082 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.39061.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38944.log.gz 2024-03-29T11:30:29.082 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.38866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39571.log 2024-03-29T11:30:29.082 INFO:teuthology.orchestra.run.smithi104.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39061.log.gz 2024-03-29T11:30:29.082 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.39412.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38866.log.gz 2024-03-29T11:30:29.082 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39334.log 2024-03-29T11:30:29.082 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40976.log 2024-03-29T11:30:29.082 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.39571.log: /var/log/ceph/ceph-client.admin.39334.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39412.log.gz 2024-03-29T11:30:29.082 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38475.log 2024-03-29T11:30:29.082 INFO:teuthology.orchestra.run.smithi104.stderr: 86.4%/var/log/ceph/ceph-client.admin.40976.log: -- replaced with /var/log/ceph/ceph-client.admin.39571.log.gz 2024-03-29T11:30:29.082 INFO:teuthology.orchestra.run.smithi104.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39334.log.gz 2024-03-29T11:30:29.082 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39100.log 2024-03-29T11:30:29.082 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.38475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39962.log 2024-03-29T11:30:29.082 INFO:teuthology.orchestra.run.smithi104.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40976.log.gz 2024-03-29T11:30:29.083 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.39100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39178.log 2024-03-29T11:30:29.083 INFO:teuthology.orchestra.run.smithi104.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38475.log.gz 2024-03-29T11:30:29.083 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.39962.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-29T11:30:29.083 INFO:teuthology.orchestra.run.smithi104.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39100.log.gz 2024-03-29T11:30:29.083 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.39178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38905.log 2024-03-29T11:30:29.083 INFO:teuthology.orchestra.run.smithi104.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39962.log.gz 2024-03-29T11:30:29.083 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38082.log 2024-03-29T11:30:29.083 INFO:teuthology.orchestra.run.smithi104.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39178.log.gz 2024-03-29T11:30:29.083 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.38905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34738.log 2024-03-29T11:30:29.083 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.38082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39493.log 2024-03-29T11:30:29.083 INFO:teuthology.orchestra.run.smithi104.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38905.log.gz 2024-03-29T11:30:29.083 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.34738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38671.log 2024-03-29T11:30:29.083 INFO:teuthology.orchestra.run.smithi104.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38082.log.gz 2024-03-29T11:30:29.083 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.39493.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28537.log 2024-03-29T11:30:29.083 INFO:teuthology.orchestra.run.smithi104.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34738.log.gz 2024-03-29T11:30:29.083 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.38671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40040.log 2024-03-29T11:30:29.084 INFO:teuthology.orchestra.run.smithi104.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39493.log.gz 2024-03-29T11:30:29.084 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph.tmp-client.admin.28537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39022.log 2024-03-29T11:30:29.084 INFO:teuthology.orchestra.run.smithi104.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38671.log.gz 2024-03-29T11:30:29.084 INFO:teuthology.orchestra.run.smithi104.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28537.log.gz 2024-03-29T11:30:29.084 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.40040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40742.log 2024-03-29T11:30:29.084 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.39022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38199.log 2024-03-29T11:30:29.084 INFO:teuthology.orchestra.run.smithi104.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40040.log.gz 2024-03-29T11:30:29.084 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.40742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39139.log 2024-03-29T11:30:29.084 INFO:teuthology.orchestra.run.smithi104.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39022.log.gz 2024-03-29T11:30:29.084 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.38199.log: gzip 85.7% -5 -- replaced with /var/log/ceph/ceph-client.admin.40742.log.gz --verbose 2024-03-29T11:30:29.084 INFO:teuthology.orchestra.run.smithi104.stderr: -- /var/log/ceph/ceph-mgr.y.log 2024-03-29T11:30:29.084 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.39139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38749.log 2024-03-29T11:30:29.084 INFO:teuthology.orchestra.run.smithi104.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.38199.log.gz 2024-03-29T11:30:29.084 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38319.log 2024-03-29T11:30:29.084 INFO:teuthology.orchestra.run.smithi104.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39139.log.gz 2024-03-29T11:30:29.085 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.38749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39610.log 2024-03-29T11:30:29.085 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.38319.log: gzip 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38749.log.gz 2024-03-29T11:30:29.085 INFO:teuthology.orchestra.run.smithi104.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39649.log 2024-03-29T11:30:29.085 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.39610.log: gzip -5 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38319.log.gz 2024-03-29T11:30:29.085 INFO:teuthology.orchestra.run.smithi104.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40157.log 2024-03-29T11:30:29.085 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.39649.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39610.log.gz 2024-03-29T11:30:29.085 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38043.log 2024-03-29T11:30:29.085 INFO:teuthology.orchestra.run.smithi104.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39649.log.gz 2024-03-29T11:30:29.085 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.40157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41250.log 2024-03-29T11:30:29.085 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5/var/log/ceph/ceph-client.admin.38043.log: --verbose 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40157.log.gz 2024-03-29T11:30:29.085 INFO:teuthology.orchestra.run.smithi104.stderr: -- /var/log/ceph/ceph-client.admin.38358.log 2024-03-29T11:30:29.085 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.41250.log: gzip -5 --verbose -- 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38043.log.gz 2024-03-29T11:30:29.085 INFO:teuthology.orchestra.run.smithi104.stderr: /var/log/ceph/ceph-client.admin.38514.log 2024-03-29T11:30:29.085 INFO:teuthology.orchestra.run.smithi104.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41250.log.gz 2024-03-29T11:30:29.085 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.38358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41094.log 2024-03-29T11:30:29.086 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5/var/log/ceph/ceph-client.admin.38514.log: --verbose 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38358.log.gz 2024-03-29T11:30:29.086 INFO:teuthology.orchestra.run.smithi104.stderr: -- /var/log/ceph/ceph-client.admin.39923.log 2024-03-29T11:30:29.086 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.41094.log: gzip -5 --verbose -- 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38514.log.gz 2024-03-29T11:30:29.086 INFO:teuthology.orchestra.run.smithi104.stderr: /var/log/ceph/ceph-osd.0.log 2024-03-29T11:30:29.086 INFO:teuthology.orchestra.run.smithi104.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41094.log.gz 2024-03-29T11:30:29.086 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.39923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39532.log 2024-03-29T11:30:29.086 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-osd.0.log: 86.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40274.log 2024-03-29T11:30:29.086 INFO:teuthology.orchestra.run.smithi104.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39923.log.gz 2024-03-29T11:30:29.086 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.39532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38121.log 2024-03-29T11:30:29.086 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.40274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39454.log 2024-03-29T11:30:29.086 INFO:teuthology.orchestra.run.smithi104.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39532.log.gz 2024-03-29T11:30:29.086 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.38121.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40274.log.gz 2024-03-29T11:30:29.086 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38121.log.gz 2024-03-29T11:30:29.086 INFO:teuthology.orchestra.run.smithi104.stderr: --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:30:29.087 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.39454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40859.log 2024-03-29T11:30:29.087 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39454.log.gz 2024-03-29T11:30:29.087 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-29T11:30:29.087 INFO:teuthology.orchestra.run.smithi104.stderr:gzip/var/log/ceph/ceph-client.admin.40859.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.40937.log 2024-03-29T11:30:29.087 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-osd.2.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40859.log.gz 2024-03-29T11:30:29.087 INFO:teuthology.orchestra.run.smithi104.stderr: 92.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:30:29.100 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-29T11:30:29.100 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34610.log 2024-03-29T11:30:29.100 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:30:29.100 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.34610.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-29T11:30:29.101 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34530.log 2024-03-29T11:30:29.101 INFO:teuthology.orchestra.run.smithi119.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34610.log.gz 2024-03-29T11:30:29.101 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-29T11:30:29.101 INFO:teuthology.orchestra.run.smithi119.stderr: 88.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:30:29.101 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.34530.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-29T11:30:29.101 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34650.log 2024-03-29T11:30:29.101 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-osd.7.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34530.log.gz 2024-03-29T11:30:29.101 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34730.log 2024-03-29T11:30:29.101 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.34650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34570.log 2024-03-29T11:30:29.101 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.34730.log: gzip -5 --verbose -- 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34650.log.gz 2024-03-29T11:30:29.101 INFO:teuthology.orchestra.run.smithi119.stderr: /var/log/ceph/ceph-client.admin.34490.log 2024-03-29T11:30:29.101 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.34570.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T11:30:29.102 INFO:teuthology.orchestra.run.smithi119.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34730.log.gz 2024-03-29T11:30:29.102 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.34490.log: 87.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.34570.log.gz -5 2024-03-29T11:30:29.102 INFO:teuthology.orchestra.run.smithi119.stderr: --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:30:29.102 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose/var/log/ceph/ceph-mgr.x.log: -- 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34490.log.gz 2024-03-29T11:30:29.102 INFO:teuthology.orchestra.run.smithi119.stderr: /var/log/ceph/ceph-mon.b.log 2024-03-29T11:30:29.102 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34450.log 2024-03-29T11:30:29.102 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34690.log 2024-03-29T11:30:29.102 INFO:teuthology.orchestra.run.smithi119.stderr: 87.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:30:29.102 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.34450.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34450.log.gz 2024-03-29T11:30:29.102 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/ceph-client.admin.34690.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34690.log.gz 2024-03-29T11:30:29.102 INFO:teuthology.orchestra.run.smithi119.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T11:30:29.104 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34344.log 2024-03-29T11:30:29.104 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.40937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40391.log 2024-03-29T11:30:29.104 INFO:teuthology.orchestra.run.smithi104.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40937.log.gz/var/log/ceph/ceph-client.admin.34344.log: 2024-03-29T11:30:29.104 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38827.log 2024-03-29T11:30:29.104 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.40391.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34344.log.gz 2024-03-29T11:30:29.104 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40313.log 2024-03-29T11:30:29.104 INFO:teuthology.orchestra.run.smithi104.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40391.log.gz 2024-03-29T11:30:29.104 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.38827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38436.log 2024-03-29T11:30:29.104 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.40313.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38827.log.gz 2024-03-29T11:30:29.104 INFO:teuthology.orchestra.run.smithi104.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40313.log.gz 2024-03-29T11:30:29.104 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41133.log 2024-03-29T11:30:29.104 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.38436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41484.log 2024-03-29T11:30:29.104 INFO:teuthology.orchestra.run.smithi104.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38436.log.gz 2024-03-29T11:30:29.105 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.41133.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41133.log.gz 2024-03-29T11:30:29.105 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40352.log 2024-03-29T11:30:29.105 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40703.log 2024-03-29T11:30:29.105 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.40352.log: /var/log/ceph/ceph-client.admin.41484.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41484.log.gz 2024-03-29T11:30:29.105 INFO:teuthology.orchestra.run.smithi104.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40352.log.gz 2024-03-29T11:30:29.105 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-29T11:30:29.105 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38397.log 2024-03-29T11:30:29.105 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.40703.log: /var/log/ceph/ceph-mon.c.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40703.log.gz 2024-03-29T11:30:29.105 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40547.log 2024-03-29T11:30:29.105 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.38397.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38397.log.gz 2024-03-29T11:30:29.105 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/ceph-client.admin.40547.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40547.log.gz 2024-03-29T11:30:29.132 INFO:teuthology.orchestra.run.smithi104.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-29T11:30:40.007 INFO:teuthology.orchestra.run.smithi119.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-29T11:30:40.544 INFO:teuthology.orchestra.run.smithi104.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-29T11:30:40.605 INFO:teuthology.orchestra.run.smithi119.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-29T11:30:40.631 INFO:teuthology.orchestra.run.smithi104.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T11:30:44.319 INFO:teuthology.orchestra.run.smithi119.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-29T11:30:44.424 INFO:teuthology.orchestra.run.smithi119.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-29T11:30:44.661 INFO:teuthology.orchestra.run.smithi104.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T11:30:44.670 INFO:teuthology.orchestra.run.smithi104.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T11:31:18.515 INFO:teuthology.orchestra.run.smithi104.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T11:31:19.251 INFO:teuthology.orchestra.run.smithi104.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-29T11:31:19.253 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-03-29T11:31:19.254 INFO:teuthology.orchestra.run.smithi104.stderr:real 0m50.223s 2024-03-29T11:31:19.254 INFO:teuthology.orchestra.run.smithi104.stderr:user 2m28.408s 2024-03-29T11:31:19.254 INFO:teuthology.orchestra.run.smithi104.stderr:sys 0m5.278s 2024-03-29T11:31:38.047 INFO:teuthology.orchestra.run.smithi119.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-29T11:31:38.049 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-03-29T11:31:38.049 INFO:teuthology.orchestra.run.smithi119.stderr:real 1m8.980s 2024-03-29T11:31:38.049 INFO:teuthology.orchestra.run.smithi119.stderr:user 1m57.588s 2024-03-29T11:31:38.049 INFO:teuthology.orchestra.run.smithi119.stderr:sys 0m4.168s 2024-03-29T11:31:38.049 INFO:tasks.ceph:Archiving logs... 2024-03-29T11:31:38.050 DEBUG:teuthology.misc:Transferring archived files from smithi104:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622681/remote/smithi104/log 2024-03-29T11:31:38.051 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:32:00.855 DEBUG:teuthology.misc:Transferring archived files from smithi119:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622681/remote/smithi119/log 2024-03-29T11:32:00.856 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:32:20.649 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:32:20.947 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=b928b3dfdb1e4e8db146bc9ee76c7658 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:32:20.951 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T11:32:21.041 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T11:32:21.113 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T11:32:21.114 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:32:21.129 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:32:21.239 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T11:32:21.252 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T11:32:21.252 DEBUG:teuthology.orchestra.run.smithi104:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:32:21.255 DEBUG:teuthology.orchestra.run.smithi119:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:32:21.270 INFO:teuthology.orchestra.run.smithi104.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:32:21.272 INFO:teuthology.orchestra.run.smithi119.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:32:21.275 INFO:teuthology.orchestra.run.smithi104.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:32:21.275 INFO:teuthology.orchestra.run.smithi104.stdout:=============================================================================== 2024-03-29T11:32:21.276 INFO:teuthology.orchestra.run.smithi104.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 23 +470us[ +467us] +/- 92ms 2024-03-29T11:32:21.276 INFO:teuthology.orchestra.run.smithi104.stdout:^* hv02.front.sepia.ceph.com 3 6 377 21 -115us[ -118us] +/- 54ms 2024-03-29T11:32:21.276 INFO:teuthology.orchestra.run.smithi104.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 23 -254us[ -257us] +/- 86ms 2024-03-29T11:32:21.276 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:32:21.277 INFO:teuthology.orchestra.run.smithi119.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:32:21.277 INFO:teuthology.orchestra.run.smithi119.stdout:=============================================================================== 2024-03-29T11:32:21.277 INFO:teuthology.orchestra.run.smithi119.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 24 +466us[ +466us] +/- 92ms 2024-03-29T11:32:21.277 INFO:teuthology.orchestra.run.smithi119.stdout:^* hv02.front.sepia.ceph.com 3 6 377 26 -107us[ -113us] +/- 54ms 2024-03-29T11:32:21.278 INFO:teuthology.orchestra.run.smithi119.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 23 -257us[ -257us] +/- 86ms 2024-03-29T11:32:21.278 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:32:21.279 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T11:32:21.293 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T11:32:21.294 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T11:32:21.308 DEBUG:teuthology.orchestra.run.smithi104:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:32:21.389 DEBUG:teuthology.orchestra.run.smithi119:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:32:21.462 DEBUG:teuthology.orchestra.run.smithi104:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:32:21.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:32:21.507 DEBUG:teuthology.orchestra.run.smithi119:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:32:21.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:32:21.539 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T11:32:21.577 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T11:32:21.597 INFO:teuthology.task.internal:Duration was 1390.565825 seconds 2024-03-29T11:32:21.598 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T11:32:21.617 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T11:32:21.617 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:32:21.621 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:32:21.666 INFO:teuthology.orchestra.run.smithi104.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:32:21.668 INFO:teuthology.orchestra.run.smithi119.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:32:21.876 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T11:32:21.876 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi104.front.sepia.ceph.com 2024-03-29T11:32:21.876 DEBUG:teuthology.orchestra.run.smithi104:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:32:21.901 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi119.front.sepia.ceph.com 2024-03-29T11:32:21.901 DEBUG:teuthology.orchestra.run.smithi119:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:32:21.935 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T11:32:21.935 DEBUG:teuthology.orchestra.run.smithi104:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:32:21.949 DEBUG:teuthology.orchestra.run.smithi119:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:32:22.013 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T11:32:22.013 DEBUG:teuthology.orchestra.run.smithi104:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:32:22.025 DEBUG:teuthology.orchestra.run.smithi119:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:32:22.150 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T11:32:22.164 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T11:32:22.165 DEBUG:teuthology.orchestra.run.smithi104:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:32:22.197 DEBUG:teuthology.orchestra.run.smithi119:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:32:22.225 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T11:32:22.238 DEBUG:teuthology.orchestra.run.smithi104:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:32:22.247 DEBUG:teuthology.orchestra.run.smithi119:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:32:22.269 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern = core 2024-03-29T11:32:22.291 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern = core 2024-03-29T11:32:22.310 DEBUG:teuthology.orchestra.run.smithi104:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:32:22.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:32:22.364 DEBUG:teuthology.orchestra.run.smithi119:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:32:22.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:32:22.380 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T11:32:22.393 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T11:32:22.396 DEBUG:teuthology.misc:Transferring archived files from smithi104:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622681/remote/smithi104 2024-03-29T11:32:22.396 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:32:22.455 DEBUG:teuthology.misc:Transferring archived files from smithi119:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622681/remote/smithi119 2024-03-29T11:32:22.455 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:32:22.514 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T11:32:22.514 DEBUG:teuthology.orchestra.run.smithi104:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:32:22.521 DEBUG:teuthology.orchestra.run.smithi119:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:32:22.553 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T11:32:22.566 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T11:32:22.566 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T11:32:22.611 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T11:32:22.612 DEBUG:teuthology.orchestra.run.smithi104:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:32:22.616 DEBUG:teuthology.orchestra.run.smithi119:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:32:22.627 INFO:teuthology.orchestra.run.smithi104.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:32 /home/ubuntu/cephtest 2024-03-29T11:32:22.631 INFO:teuthology.orchestra.run.smithi119.stdout: 265819 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:32 /home/ubuntu/cephtest 2024-03-29T11:32:22.633 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T11:32:22.649 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T11:32:22.683 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T11:32:22.711 DEBUG:teuthology.nuke:shortname: smithi104 2024-03-29T11:32:22.712 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:32:22.740 DEBUG:teuthology.nuke:shortname: smithi119 2024-03-29T11:32:22.741 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:32:22.746 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi104.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622681', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:01:12.759016', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMapDgIsdLYjW2H7cUSLyAA2wd86CeL/sKuJHvyFFC9a+OL//CvJkSpgxOVDA9BOb4KTT7tX67mxUCkvnffYm4g='} 2024-03-29T11:32:22.773 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi119.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622681', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:01:12.757536', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM6RWsxLIJXsXGfKg2EOoww8oIOk8ov7+dbvExTnnCoD8eyyxNhyxmQFEDeoq7WrTYD4l6S+hXD66rx7Qe4rA34='} 2024-03-29T11:32:22.802 INFO:teuthology.orchestra.console.smithi104:Power off 2024-03-29T11:32:22.803 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:32:22.830 DEBUG:teuthology.orchestra.console.smithi104:power off output: Chassis Power Control: Down/Off 2024-03-29T11:32:22.831 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:32:22.854 INFO:teuthology.orchestra.console.smithi119:Power off 2024-03-29T11:32:22.855 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:32:22.867 DEBUG:teuthology.orchestra.console.smithi104:check power output: Chassis Power is on 2024-03-29T11:32:22.880 DEBUG:teuthology.orchestra.console.smithi119:power off output: Chassis Power Control: Down/Off 2024-03-29T11:32:22.880 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:32:22.906 DEBUG:teuthology.orchestra.console.smithi119:check power output: Chassis Power is on 2024-03-29T11:32:26.869 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:32:26.908 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:32:26.982 DEBUG:teuthology.orchestra.console.smithi104:check power output: Chassis Power is on 2024-03-29T11:32:27.018 DEBUG:teuthology.orchestra.console.smithi119:check power output: Chassis Power is on 2024-03-29T11:32:30.986 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:32:31.019 DEBUG:teuthology.orchestra.console.smithi119:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:32:31.099 DEBUG:teuthology.orchestra.console.smithi104:check power output: Chassis Power is off 2024-03-29T11:32:31.131 DEBUG:teuthology.orchestra.console.smithi119:check power output: Chassis Power is off 2024-03-29T11:32:31.199 INFO:teuthology.orchestra.console.smithi104:Power off completed 2024-03-29T11:32:31.232 INFO:teuthology.orchestra.console.smithi119:Power off completed 2024-03-29T11:32:31.337 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} duration: 1390.5658252239227 failure_reason: reached maximum tries (91) after waiting for 540 seconds flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=b928b3dfdb1e4e8db146bc9ee76c7658 status: fail success: false 2024-03-29T11:32:31.338 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:32:31.424 INFO:teuthology.run:FAIL